Sep 11 19:58:29.796380 ip-10-0-128-209 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 19:58:29.796539 ip-10-0-128-209 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 19:58:29.796629 ip-10-0-128-209 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 19:58:29.797102 ip-10-0-128-209 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 19:58:39.834764 ip-10-0-128-209 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 19:58:39.834779 ip-10-0-128-209 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 7b21dbf645e14c1eb2e08513607b50f0 --
Sep 11 20:00:29.898804 ip-10-0-128-209 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 20:00:30.251130 ip-10-0-128-209 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 20:00:30.251130 ip-10-0-128-209 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 20:00:30.251130 ip-10-0-128-209 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 20:00:30.251130 ip-10-0-128-209 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 20:00:30.251130 ip-10-0-128-209 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 20:00:30.251844 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.251192    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 20:00:30.255242 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255219    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 20:00:30.255242 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255239    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 20:00:30.255242 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255243    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 20:00:30.255242 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255247    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 20:00:30.255242 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255250    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 20:00:30.255433 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255253    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 20:00:30.255433 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255257    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 20:00:30.255433 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255259    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 20:00:30.255433 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255262    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 20:00:30.255433 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255265    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 20:00:30.255433 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255268    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 20:00:30.255433 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255270    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 20:00:30.255433 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255273    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 20:00:30.255433 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255276    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 20:00:30.255433 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255280    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 20:00:30.255433 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255284    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 20:00:30.255433 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255288    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 20:00:30.255433 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255291    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 20:00:30.255433 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255294    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 20:00:30.255433 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255297    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 20:00:30.255433 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255299    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 20:00:30.255433 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255302    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 20:00:30.255433 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255306    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 20:00:30.255433 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255309    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 20:00:30.255899 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255312    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 20:00:30.255899 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255315    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 20:00:30.255899 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255317    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 20:00:30.255899 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255320    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 20:00:30.255899 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255323    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 20:00:30.255899 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255325    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 20:00:30.255899 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255328    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 20:00:30.255899 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255338    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 20:00:30.255899 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255340    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 20:00:30.255899 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255343    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 20:00:30.255899 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255345    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 20:00:30.255899 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255348    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 20:00:30.255899 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255350    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 20:00:30.255899 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255353    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 20:00:30.255899 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255356    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 20:00:30.255899 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255359    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 20:00:30.255899 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255361    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 20:00:30.255899 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255364    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 20:00:30.255899 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255366    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 20:00:30.255899 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255369    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 20:00:30.256388 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255372    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 20:00:30.256388 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255374    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 20:00:30.256388 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255377    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 20:00:30.256388 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255379    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 20:00:30.256388 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255382    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 20:00:30.256388 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255385    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 20:00:30.256388 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255388    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 20:00:30.256388 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255391    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 20:00:30.256388 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255394    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 20:00:30.256388 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255397    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 20:00:30.256388 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255400    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 20:00:30.256388 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255402    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 20:00:30.256388 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255405    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 20:00:30.256388 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255408    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 20:00:30.256388 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255410    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 20:00:30.256388 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255413    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 20:00:30.256388 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255415    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 20:00:30.256388 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255418    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 20:00:30.256388 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255420    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 20:00:30.256388 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255424    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 20:00:30.256890 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255426    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 20:00:30.256890 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255430    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 20:00:30.256890 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255434    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 20:00:30.256890 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255437    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 20:00:30.256890 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255441    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 20:00:30.256890 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255444    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 20:00:30.256890 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255447    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 20:00:30.256890 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255449    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 20:00:30.256890 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255452    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 20:00:30.256890 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255455    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 20:00:30.256890 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255458    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 20:00:30.256890 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255461    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 20:00:30.256890 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255463    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 20:00:30.256890 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255465    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 20:00:30.256890 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255468    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 20:00:30.256890 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255471    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 20:00:30.256890 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255474    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 20:00:30.256890 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255476    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 20:00:30.256890 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255479    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 20:00:30.257349 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255482    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 20:00:30.257349 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255485    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 20:00:30.257349 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255487    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 20:00:30.257349 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255501    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 20:00:30.257349 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255503    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 20:00:30.257349 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255506    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 20:00:30.257349 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255509    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 20:00:30.257349 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255512    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 20:00:30.257349 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255514    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 20:00:30.257349 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255517    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 20:00:30.257349 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255519    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 20:00:30.257349 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255522    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 20:00:30.257349 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255525    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 20:00:30.257349 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255528    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 20:00:30.257349 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255531    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 20:00:30.257349 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.255533    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 20:00:30.257349 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.256961    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 20:00:30.257349 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.256971    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 20:00:30.257349 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.256974    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 20:00:30.257349 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.256977    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 20:00:30.257837 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.256980    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 20:00:30.257837 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.256983    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 20:00:30.257837 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.256986    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 20:00:30.257837 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.256988    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 20:00:30.257837 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.256991    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 20:00:30.257837 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.256994    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 20:00:30.257837 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.256997    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 20:00:30.257837 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.256999    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 20:00:30.257837 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257002    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 20:00:30.257837 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257006    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 20:00:30.257837 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257010    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 20:00:30.257837 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257012    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 20:00:30.257837 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257015    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 20:00:30.257837 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257018    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 20:00:30.257837 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257021    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 20:00:30.257837 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257024    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 20:00:30.257837 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257027    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 20:00:30.257837 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257029    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 20:00:30.257837 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257032    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 20:00:30.257837 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257035    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 20:00:30.258320 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257037    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 20:00:30.258320 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257040    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 20:00:30.258320 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257042    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 20:00:30.258320 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257045    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 20:00:30.258320 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257048    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 20:00:30.258320 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257050    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 20:00:30.258320 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257053    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 20:00:30.258320 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257057    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 20:00:30.258320 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257060    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 20:00:30.258320 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257064    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 20:00:30.258320 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257067    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 20:00:30.258320 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257070    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 20:00:30.258320 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257073    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 20:00:30.258320 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257076    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 20:00:30.258320 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257078    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 20:00:30.258320 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257081    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 20:00:30.258320 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257084    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 20:00:30.258320 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257086    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 20:00:30.258320 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257089    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 20:00:30.258320 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257092    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 20:00:30.258820 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257094    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 20:00:30.258820 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257097    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 20:00:30.258820 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257099    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 20:00:30.258820 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257102    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 20:00:30.258820 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257105    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 20:00:30.258820 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257107    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 20:00:30.258820 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257111    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 20:00:30.258820 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257114    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 20:00:30.258820 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257117    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 20:00:30.258820 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257119    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 20:00:30.258820 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257122    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 20:00:30.258820 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257124    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 20:00:30.258820 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257127    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 20:00:30.258820 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257130    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 20:00:30.258820 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257132    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 20:00:30.258820 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257136    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 20:00:30.258820 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257138    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 20:00:30.258820 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257140    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 20:00:30.258820 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257143    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 20:00:30.258820 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257145    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 20:00:30.259307 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257148    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 20:00:30.259307 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257150    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 20:00:30.259307 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257153    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 20:00:30.259307 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257156    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 20:00:30.259307 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257158    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 20:00:30.259307 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257161    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 20:00:30.259307 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257163    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 20:00:30.259307 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257167    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 20:00:30.259307 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257170    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 20:00:30.259307 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257173    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 20:00:30.259307 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257176    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 20:00:30.259307 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257179    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 20:00:30.259307 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257181    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 20:00:30.259307 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257184    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 20:00:30.259307 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257188    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 20:00:30.259307 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257191    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 20:00:30.259307 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257194    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 20:00:30.259307 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257197    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 20:00:30.259307 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257200    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 20:00:30.259859 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257203    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 20:00:30.259859 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257206    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 20:00:30.259859 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257209    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 20:00:30.259859 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257211    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 20:00:30.259859 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257214    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 20:00:30.259859 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257217    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 20:00:30.259859 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257219    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 20:00:30.259859 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257223    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 20:00:30.259859 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257225    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 20:00:30.259859 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257228    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 20:00:30.259859 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257230    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 20:00:30.259859 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257233    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 20:00:30.259859 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257235    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 20:00:30.259859 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257238    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 20:00:30.259859 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257240    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 20:00:30.259859 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257243    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 20:00:30.259859 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257620    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 20:00:30.259859 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257625    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 20:00:30.259859 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257628    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 20:00:30.259859 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257631    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 20:00:30.260354 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257633    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 20:00:30.260354 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257636    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 20:00:30.260354 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257639    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 20:00:30.260354 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257642    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 20:00:30.260354 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257645    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 20:00:30.260354 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257647    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 20:00:30.260354 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257650    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 20:00:30.260354 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257652    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 20:00:30.260354 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257655    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 20:00:30.260354 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257658    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 20:00:30.260354 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257661    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 20:00:30.260354 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257664    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 20:00:30.260354 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257667    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 20:00:30.260354 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257670    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 20:00:30.260354 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257673    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 20:00:30.260354 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257675    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 20:00:30.260354 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257678    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 20:00:30.260354 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257680    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 20:00:30.260354 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257683    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 20:00:30.260354 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257686    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 20:00:30.260850 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257688    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 20:00:30.260850 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257691    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 20:00:30.260850 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257693    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 20:00:30.260850 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257696    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 20:00:30.260850 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257698    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 20:00:30.260850 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257701    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 20:00:30.260850 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257703    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 20:00:30.260850 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257706    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 20:00:30.260850 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257709    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 20:00:30.260850 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257712    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 20:00:30.260850 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257715    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 20:00:30.260850 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257717    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 20:00:30.260850 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257720    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 20:00:30.260850 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257723    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 20:00:30.260850 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257725    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 20:00:30.260850 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257728    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 20:00:30.260850 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257731    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 20:00:30.260850 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257733    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 20:00:30.260850 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257735    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 20:00:30.260850 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257738    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 20:00:30.261335 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257741    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 20:00:30.261335 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257743    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 20:00:30.261335 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257746    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 20:00:30.261335 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257748    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 20:00:30.261335 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257751    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 20:00:30.261335 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257754    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 20:00:30.261335 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257757    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 20:00:30.261335 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257759    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 20:00:30.261335 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257762    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 20:00:30.261335 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257765    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 20:00:30.261335 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257769    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 20:00:30.261335 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257773    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 20:00:30.261335 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257775    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 20:00:30.261335 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257778    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 20:00:30.261335 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257781    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 20:00:30.261335 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257784    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 20:00:30.261335 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257786    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 20:00:30.261335 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257789    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 20:00:30.261335 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257792    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 20:00:30.261834 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257794    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 20:00:30.261834 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257797    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 20:00:30.261834 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257799    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 20:00:30.261834 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257802    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 20:00:30.261834 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257805    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 20:00:30.261834 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257807    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 20:00:30.261834 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257810    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 20:00:30.261834 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257812    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 20:00:30.261834 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257815    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 20:00:30.261834 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257817    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 20:00:30.261834 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257820    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 20:00:30.261834 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257822    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 20:00:30.261834 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257825    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 20:00:30.261834 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257827    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 20:00:30.261834 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257830    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 20:00:30.261834 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257834    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 20:00:30.261834 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257837    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 20:00:30.261834 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257840    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 20:00:30.261834 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257842    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 20:00:30.261834 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257846    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 20:00:30.261834 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257848    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 20:00:30.262336 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257853    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 20:00:30.262336 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257856    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 20:00:30.262336 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257859    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 20:00:30.262336 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257862    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 20:00:30.262336 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257865    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 20:00:30.262336 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257867    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 20:00:30.262336 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257870    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 20:00:30.262336 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257873    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 20:00:30.262336 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257875    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 20:00:30.262336 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257878    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 20:00:30.262336 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257882    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 20:00:30.262336 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257885    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 20:00:30.262336 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257887    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 20:00:30.262336 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257890    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 20:00:30.262336 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.257894    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 20:00:30.262336 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.257965    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 20:00:30.262336 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258008    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 20:00:30.262336 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258016    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 20:00:30.262336 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258020    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 20:00:30.262336 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258024    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 20:00:30.262825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258028    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 20:00:30.262825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258032    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 20:00:30.262825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258036    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 20:00:30.262825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258039    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 20:00:30.262825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258042    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 20:00:30.262825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258046    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 20:00:30.262825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258049    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 20:00:30.262825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258052    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 20:00:30.262825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258055    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 11 20:00:30.262825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258058    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 20:00:30.262825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258061    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 11 20:00:30.262825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258065    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 20:00:30.262825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258068    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 20:00:30.262825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258074    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 11 20:00:30.262825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258077    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 20:00:30.262825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258080    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 20:00:30.262825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258083    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 20:00:30.262825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258086    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 20:00:30.262825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258094    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 20:00:30.262825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258097    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 20:00:30.262825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258100    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 20:00:30.262825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258104    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 20:00:30.262825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258108    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 20:00:30.262825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258111    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258115    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258118    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258121    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258125    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258128    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258131    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258135    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258138    2580 flags.go:64] FLAG: --enable-server="true"
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258141    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258145    2580 flags.go:64] FLAG: --event-burst="100"
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258149    2580 flags.go:64] FLAG: --event-qps="50"
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258151    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258154    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258157    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258161    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258164    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258167    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258170    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258173    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258176    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258179    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258182    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258185    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258188    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 20:00:30.263449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258191    2580 flags.go:64] FLAG: --feature-gates=""
Sep 11 20:00:30.264070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258194    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 20:00:30.264070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258198    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 20:00:30.264070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258201    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 20:00:30.264070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258204    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 20:00:30.264070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258207    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 20:00:30.264070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258211    2580 flags.go:64] FLAG: --help="false"
Sep 11 20:00:30.264070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258214    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-128-209.ec2.internal"
Sep 11 20:00:30.264070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258217    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 20:00:30.264070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258220    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 20:00:30.264070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258223    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 20:00:30.264070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258226    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 20:00:30.264070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258230    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 20:00:30.264070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258232    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 20:00:30.264070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258237    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 20:00:30.264070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258240    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 20:00:30.264070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258258    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 20:00:30.264070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258262    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 20:00:30.264070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258265    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 20:00:30.264070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258269    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 11 20:00:30.264070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258272    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 20:00:30.264070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258275    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 20:00:30.264070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258279    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 20:00:30.264070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258281    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 20:00:30.264070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258284    2580 flags.go:64] FLAG: --lock-file=""
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258287    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258290    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258293    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258299    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258303    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258306    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258308    2580 flags.go:64] FLAG: --logging-format="text"
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258311    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258315    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258318    2580 flags.go:64] FLAG: --manifest-url=""
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258321    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258325    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258329    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258334    2580 flags.go:64] FLAG: --max-pods="110"
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258337    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258340    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258343    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258346    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258349    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258352    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258355    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258363    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258367    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258370    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 20:00:30.264669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258372    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258375    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258380    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258383    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258386    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258389    2580 flags.go:64] FLAG: --port="10250"
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258392    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258395    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-025a1b2149b2983c2"
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258399    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258401    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258404    2580 flags.go:64] FLAG: --register-node="true"
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258407    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258411    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258414    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258417    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258420    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258423    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258426    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258429    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258432    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258434    2580 flags.go:64] FLAG: --runonce="false"
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258437    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258441    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258444    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258447    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 20:00:30.265275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258450    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258454    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258457    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258460    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258465    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258468    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258471    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258473    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258476    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258479    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258482    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258487    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258501    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258504    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258509    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258512    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258514    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258517    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258520    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258523    2580 flags.go:64] FLAG: --v="2"
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258528    2580 flags.go:64] FLAG: --version="false"
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258532    2580 flags.go:64] FLAG: --vmodule=""
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258536    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.258539    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258655    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 20:00:30.265888 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258660    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 20:00:30.266487 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258664    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 20:00:30.266487 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258666    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 20:00:30.266487 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258671    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 20:00:30.266487 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258676    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 20:00:30.266487 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258679    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 20:00:30.266487 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258683    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 20:00:30.266487 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258687    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 20:00:30.266487 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258691    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 20:00:30.266487 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258694    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 20:00:30.266487 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258697    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 20:00:30.266487 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258701    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 20:00:30.266487 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258704    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 20:00:30.266487 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258707    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 20:00:30.266487 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258710    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 20:00:30.266487 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258713    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 20:00:30.266487 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258715    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 20:00:30.266487 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258718    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 20:00:30.266487 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258721    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 20:00:30.266487 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258724    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 20:00:30.266992 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258727    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 20:00:30.266992 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258729    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 20:00:30.266992 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258732    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 20:00:30.266992 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258735    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 20:00:30.266992 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258737    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 20:00:30.266992 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258740    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 20:00:30.266992 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258743    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 20:00:30.266992 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258746    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 20:00:30.266992 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258749    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 20:00:30.266992 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258758    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 20:00:30.266992 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258761    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 20:00:30.266992 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258764    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 20:00:30.266992 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258767    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 20:00:30.266992 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258770    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 20:00:30.266992 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258773    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 20:00:30.266992 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258776    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 20:00:30.266992 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258780    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 20:00:30.266992 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258783    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 20:00:30.266992 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258785    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 20:00:30.266992 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258788    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 20:00:30.267511 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258791    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 20:00:30.267511 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258794    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 20:00:30.267511 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258796    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 20:00:30.267511 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258800    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 20:00:30.267511 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258803    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 20:00:30.267511 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258805    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 20:00:30.267511 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258808    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 20:00:30.267511 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258810    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 20:00:30.267511 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258813    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 20:00:30.267511 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258816    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 20:00:30.267511 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258818    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 20:00:30.267511 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258821    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 20:00:30.267511 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258823    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 20:00:30.267511 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258826    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 20:00:30.267511 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258829    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 20:00:30.267511 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258831    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 20:00:30.267511 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258834    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 20:00:30.267511 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258837    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 20:00:30.267511 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258840    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 20:00:30.267965 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258843    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 20:00:30.267965 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258845    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 20:00:30.267965 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258848    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 20:00:30.267965 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258851    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 20:00:30.267965 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258854    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 20:00:30.267965 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258856    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 20:00:30.267965 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258859    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 20:00:30.267965 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258861    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 20:00:30.267965 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258864    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 20:00:30.267965 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258868    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 20:00:30.267965 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258870    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 20:00:30.267965 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258873    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 20:00:30.267965 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258876    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 20:00:30.267965 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258878    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 20:00:30.267965 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258881    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 20:00:30.267965 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258883    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 20:00:30.267965 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258887    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 20:00:30.267965 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258889    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 20:00:30.267965 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258892    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 20:00:30.267965 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258894    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 20:00:30.268457 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258897    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 20:00:30.268457 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258900    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 20:00:30.268457 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258902    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 20:00:30.268457 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258905    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 20:00:30.268457 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258907    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 20:00:30.268457 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258910    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 20:00:30.268457 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258913    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 20:00:30.268457 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258915    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 20:00:30.268457 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258918    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 20:00:30.268457 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258920    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 20:00:30.268457 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258923    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 20:00:30.268457 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258925    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 20:00:30.268457 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258928    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 20:00:30.268457 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258930    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 20:00:30.268457 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258933    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 20:00:30.268457 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258936    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 20:00:30.268457 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258939    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 20:00:30.268457 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258942    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 20:00:30.268457 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.258944    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 20:00:30.268937 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.259565    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 20:00:30.268937 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.266826    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 20:00:30.268937 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.266842    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 20:00:30.268937 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.267508    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 20:00:30.268937 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.267515    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 20:00:30.268937 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267578    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 20:00:30.268937 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267583    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 20:00:30.268937 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267586    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 20:00:30.268937 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267590    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 20:00:30.268937 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267593    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 20:00:30.268937 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267596    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 20:00:30.268937 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267599    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 20:00:30.268937 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267601    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 20:00:30.268937 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267604    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 20:00:30.268937 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267607    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 20:00:30.269316 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267609    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 20:00:30.269316 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267612    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 20:00:30.269316 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267614    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 20:00:30.269316 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267617    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 20:00:30.269316 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267620    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 20:00:30.269316 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267623    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 20:00:30.269316 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267626    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 20:00:30.269316 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267628    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 20:00:30.269316 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267631    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 20:00:30.269316 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267634    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 20:00:30.269316 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267636    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 20:00:30.269316 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267639    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 20:00:30.269316 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267641    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 20:00:30.269316 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267644    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 20:00:30.269316 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267647    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 20:00:30.269316 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267649    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 20:00:30.269316 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267652    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 20:00:30.269316 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267654    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 20:00:30.269316 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267657    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 20:00:30.269316 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267661    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 20:00:30.269801 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267664    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 20:00:30.269801 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267667    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 20:00:30.269801 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267669    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 20:00:30.269801 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267672    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 20:00:30.269801 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267675    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 20:00:30.269801 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267678    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 20:00:30.269801 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267681    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 20:00:30.269801 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267684    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 20:00:30.269801 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267686    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 20:00:30.269801 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267689    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 20:00:30.269801 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267693    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 20:00:30.269801 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267697    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 20:00:30.269801 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267700    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 20:00:30.269801 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267703    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 20:00:30.269801 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267705    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 20:00:30.269801 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267709    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 20:00:30.269801 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267712    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 20:00:30.269801 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267714    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 20:00:30.269801 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267718    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 20:00:30.270311 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267722    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 20:00:30.270311 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267725    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 20:00:30.270311 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267728    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 20:00:30.270311 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267730    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 20:00:30.270311 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267733    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 20:00:30.270311 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267736    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 20:00:30.270311 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267739    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 20:00:30.270311 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267742    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 20:00:30.270311 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267744    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 20:00:30.270311 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267747    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 20:00:30.270311 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267750    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 20:00:30.270311 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267752    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 20:00:30.270311 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267756    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 20:00:30.270311 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267762    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 20:00:30.270311 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267765    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 20:00:30.270311 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267769    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 20:00:30.270311 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267772    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 20:00:30.270311 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267775    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 20:00:30.270311 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267778    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 20:00:30.270789 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267781    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 20:00:30.270789 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267784    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 20:00:30.270789 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267786    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 20:00:30.270789 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267789    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 20:00:30.270789 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267792    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 20:00:30.270789 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267794    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 20:00:30.270789 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267797    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 20:00:30.270789 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267799    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 20:00:30.270789 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267802    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 20:00:30.270789 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267804    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 20:00:30.270789 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267807    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 20:00:30.270789 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267810    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 20:00:30.270789 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267812    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 20:00:30.270789 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267815    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 20:00:30.270789 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267817    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 20:00:30.270789 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267820    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 20:00:30.270789 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267822    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 20:00:30.270789 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267825    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 20:00:30.270789 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267827    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 20:00:30.271248 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267830    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 20:00:30.271248 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267833    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 20:00:30.271248 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267835    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 20:00:30.271248 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267838    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 20:00:30.271248 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267840    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 20:00:30.271248 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267843    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 20:00:30.271248 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267847    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 20:00:30.271248 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267850    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 20:00:30.271248 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267853    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 20:00:30.271248 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267855    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 20:00:30.271248 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267858    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 20:00:30.271248 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.267860    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 20:00:30.271248 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.267865    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 20:00:30.271248 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268009    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 20:00:30.271248 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268014    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 20:00:30.271691 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268017    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 20:00:30.271691 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268020    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 20:00:30.271691 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268022    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 20:00:30.271691 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268025    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 20:00:30.271691 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268028    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 20:00:30.271691 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268031    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 20:00:30.271691 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268034    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 20:00:30.271691 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268036    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 20:00:30.271691 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268056    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 20:00:30.271691 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268060    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 20:00:30.271691 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268063    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 20:00:30.271691 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268066    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 20:00:30.271691 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268069    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 20:00:30.271691 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268072    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 20:00:30.271691 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268074    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 20:00:30.271691 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268077    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 20:00:30.271691 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268080    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 20:00:30.271691 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268082    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 20:00:30.271691 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268085    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 20:00:30.271691 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268088    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 20:00:30.272167 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268090    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 20:00:30.272167 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268093    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 20:00:30.272167 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268096    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 20:00:30.272167 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268099    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 20:00:30.272167 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268101    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 20:00:30.272167 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268104    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 20:00:30.272167 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268112    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 20:00:30.272167 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268115    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 20:00:30.272167 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268118    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 20:00:30.272167 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268121    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 20:00:30.272167 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268123    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 20:00:30.272167 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268126    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 20:00:30.272167 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268129    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 20:00:30.272167 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268131    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 20:00:30.272167 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268134    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 20:00:30.272167 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268136    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 20:00:30.272167 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268139    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 20:00:30.272167 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268141    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 20:00:30.272167 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268145    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 20:00:30.272167 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268149    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 20:00:30.272649 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268152    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 20:00:30.272649 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268155    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 20:00:30.272649 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268158    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 20:00:30.272649 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268161    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 20:00:30.272649 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268164    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 20:00:30.272649 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268166    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 20:00:30.272649 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268169    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 20:00:30.272649 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268172    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 20:00:30.272649 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268175    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 20:00:30.272649 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268178    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 20:00:30.272649 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268181    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 20:00:30.272649 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268185    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 20:00:30.272649 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268188    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 20:00:30.272649 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268191    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 20:00:30.272649 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268194    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 20:00:30.272649 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268197    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 20:00:30.272649 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268200    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 20:00:30.272649 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268203    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 20:00:30.272649 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268205    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 20:00:30.273095 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268214    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 20:00:30.273095 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268217    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 20:00:30.273095 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268219    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 20:00:30.273095 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268222    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 20:00:30.273095 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268225    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 20:00:30.273095 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268227    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 20:00:30.273095 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268230    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 20:00:30.273095 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268232    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 20:00:30.273095 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268235    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 20:00:30.273095 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268238    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 20:00:30.273095 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268240    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 20:00:30.273095 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268243    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 20:00:30.273095 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268245    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 20:00:30.273095 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268248    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 20:00:30.273095 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268251    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 20:00:30.273095 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268253    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 20:00:30.273095 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268256    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 20:00:30.273095 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268258    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 20:00:30.273095 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268261    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 20:00:30.273555 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268264    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 20:00:30.273555 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268266    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 20:00:30.273555 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268269    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 20:00:30.273555 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268271    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 20:00:30.273555 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268274    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 20:00:30.273555 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268276    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 20:00:30.273555 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268279    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 20:00:30.273555 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268283    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 20:00:30.273555 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268286    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 20:00:30.273555 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268290    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 20:00:30.273555 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268293    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 20:00:30.273555 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268296    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 20:00:30.273555 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268299    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 20:00:30.273555 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268302    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 20:00:30.273555 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268310    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 20:00:30.273555 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268313    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 20:00:30.273555 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268316    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 20:00:30.273555 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268319    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 20:00:30.273555 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:30.268322    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 20:00:30.274050 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.268327    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 20:00:30.274050 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.268435    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 20:00:30.274050 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.270728    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 20:00:30.274050 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.271656    2580 server.go:1019] "Starting client certificate rotation"
Sep 11 20:00:30.274050 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.271772    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 20:00:30.274050 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.271839    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 20:00:30.274338 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.274322    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 20:00:30.279666 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.279641    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 20:00:30.289470 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.289454    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 11 20:00:30.292148 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.292135    2580 log.go:25] "Validated CRI v1 image API"
Sep 11 20:00:30.292985 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.292968    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 20:00:30.296879 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.296853    2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7e5e0bca-33e4-4bbc-ba41-b41e5ac4d16d:/dev/nvme0n1p3 cae4bd68-a0cc-426a-8f2c-1da99f8bfac6:/dev/nvme0n1p4]
Sep 11 20:00:30.296943 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.296876    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 20:00:30.298649 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.298628    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 20:00:30.302262 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.302157    2580 manager.go:216] Machine: {Timestamp:2026-09-11 20:00:30.300914039 +0000 UTC m=+0.293076300 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3150559 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fa7de49d3fa270e6c7a9a6c890dc6 SystemUUID:ec2fa7de-49d3-fa27-0e6c-7a9a6c890dc6 BootID:7b21dbf6-45e1-4c1e-b2e0-8513607b50f0 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0a:e2:6d:58:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0a:e2:6d:58:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:fd:41:9a:57:cc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 20:00:30.302262 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.302257    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 20:00:30.302380 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.302336    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 20:00:30.305701 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.305671    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 20:00:30.305835 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.305704    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-209.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 20:00:30.307014 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.307003    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 20:00:30.307047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.307017    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 20:00:30.307047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.307024    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 20:00:30.307632 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.307621    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 20:00:30.308663 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.308654    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 20:00:30.311966 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.311954    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 20:00:30.312089 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.312080    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 20:00:30.312124 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.312109    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 20:00:30.312124 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.312122    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 20:00:30.312177 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.312147    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 20:00:30.312177 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.312158    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 11 20:00:30.312177 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.312167    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 20:00:30.312562 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.312550    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 20:00:30.312602 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.312569    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 20:00:30.314778 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.314759    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 20:00:30.317464 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.317449    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 20:00:30.317561 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.317506    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 20:00:30.318836 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.318819    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xkfnn"
Sep 11 20:00:30.319993 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.319976    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 20:00:30.320231 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:30.320197    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-209.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 20:00:30.320231 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:30.320199    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 20:00:30.320961 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.320940    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 20:00:30.321032 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.320966    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 20:00:30.321032 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.320977    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 20:00:30.321032 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.320992    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 20:00:30.321032 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.321003    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 20:00:30.321032 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.321023    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 20:00:30.321032 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.321032    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 20:00:30.321284 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.321044    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 20:00:30.321284 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.321059    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 20:00:30.321284 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.321071    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 20:00:30.321284 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.321080    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 20:00:30.322556 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.322540    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 20:00:30.322618 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.322568    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 20:00:30.324121 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.324096    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xkfnn"
Sep 11 20:00:30.325871 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.325856    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-209.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 20:00:30.327935 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.327921    2580 server.go:1268] "Started kubelet"
Sep 11 20:00:30.328051 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.328023    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 20:00:30.328051 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.328017    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 20:00:30.328171 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.328080    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 20:00:30.328960 ip-10-0-128-209 systemd[1]: Started Kubernetes Kubelet.
Sep 11 20:00:30.330440 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.330422    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 20:00:30.330669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.330589    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 20:00:30.332431 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:30.332406    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 20:00:30.335087 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.335071    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 20:00:30.335181 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.335092    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 20:00:30.335267 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:30.335248    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-209.ec2.internal\" not found"
Sep 11 20:00:30.335680 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.335661    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 20:00:30.335680 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.335679    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 20:00:30.335828 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.335681    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 20:00:30.335828 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.335784    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 20:00:30.335828 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.335794    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 20:00:30.336085 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.336066    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 20:00:30.336085 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.336086    2580 factory.go:55] Registering systemd factory
Sep 11 20:00:30.336212 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.336095    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 11 20:00:30.336313 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.336302    2580 factory.go:153] Registering CRI-O factory
Sep 11 20:00:30.336358 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.336317    2580 factory.go:223] Registration of the crio container factory successfully
Sep 11 20:00:30.336358 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.336353    2580 factory.go:103] Registering Raw factory
Sep 11 20:00:30.336441 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.336368    2580 manager.go:1196] Started watching for new ooms in manager
Sep 11 20:00:30.336711 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.336700    2580 manager.go:349] Starting recovery of all containers
Sep 11 20:00:30.337375 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.337355    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 20:00:30.340890 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:30.340866    2580 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-209.ec2.internal\" not found" node="ip-10-0-128-209.ec2.internal"
Sep 11 20:00:30.347806 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.347791    2580 manager.go:354] Recovery completed
Sep 11 20:00:30.353478 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.353439    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 20:00:30.355171 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.355154    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 20:00:30.355266 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.355190    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 20:00:30.355266 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.355204    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-209.ec2.internal" event="NodeHasSufficientPID"
Sep 11 20:00:30.355923 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.355910    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 20:00:30.355923 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.355922    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 20:00:30.356023 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.355928    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 20:00:30.356023 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.355939    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 20:00:30.356723 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.356706    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 20:00:30.356776 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.356739    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 20:00:30.356776 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.356752    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-209.ec2.internal" event="NodeHasSufficientPID"
Sep 11 20:00:30.357200 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.357186    2580 policy_none.go:49] "None policy: Start"
Sep 11 20:00:30.357200 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.357202    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 20:00:30.357301 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.357211    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 20:00:30.358177 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.358167    2580 policy_none.go:47] "Start"
Sep 11 20:00:30.380106 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.380085    2580 manager.go:337] "Starting Device Plugin manager"
Sep 11 20:00:30.380208 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:30.380138    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 20:00:30.380208 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.380153    2580 server.go:87] "Starting device plugin registration server"
Sep 11 20:00:30.380468 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.380454    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 20:00:30.380546 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.380468    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 20:00:30.380592 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.380571    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 20:00:30.380646 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.380640    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 20:00:30.380697 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.380648    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 20:00:30.381289 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:30.381266    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 20:00:30.381370 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:30.381326    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-209.ec2.internal\" not found"
Sep 11 20:00:30.432981 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.432954    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 20:00:30.434088 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.434074    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 20:00:30.434172 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.434131    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 20:00:30.434172 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.434153    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 20:00:30.434287 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:30.434251    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 20:00:30.436787 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.436770    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 20:00:30.480833 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.480809    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 20:00:30.481719 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.481702    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 20:00:30.481810 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.481734    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 20:00:30.481810 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.481747    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-209.ec2.internal" event="NodeHasSufficientPID"
Sep 11 20:00:30.481810 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.481775    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-209.ec2.internal"
Sep 11 20:00:30.492672 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.492656    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-209.ec2.internal"
Sep 11 20:00:30.492735 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:30.492675    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-209.ec2.internal\": node \"ip-10-0-128-209.ec2.internal\" not found"
Sep 11 20:00:30.526058 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:30.525980    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-209.ec2.internal\" not found"
Sep 11 20:00:30.535060 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.535036    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-209.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-209.ec2.internal"]
Sep 11 20:00:30.535160 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.535108    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 20:00:30.536679 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.536664    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 20:00:30.536759 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.536690    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 20:00:30.536759 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.536699    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-209.ec2.internal" event="NodeHasSufficientPID"
Sep 11 20:00:30.537812 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.537800    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 20:00:30.537958 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.537944    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-209.ec2.internal"
Sep 11 20:00:30.538021 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.537977    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 20:00:30.538538 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.538509    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 20:00:30.538631 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.538551    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 20:00:30.538631 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.538561    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-209.ec2.internal" event="NodeHasSufficientPID"
Sep 11 20:00:30.538631 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.538523    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 20:00:30.538631 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.538629    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 20:00:30.538771 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.538643    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-209.ec2.internal" event="NodeHasSufficientPID"
Sep 11 20:00:30.539964 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.539951    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-209.ec2.internal"
Sep 11 20:00:30.540018 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.539974    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 20:00:30.540668 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.540651    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 20:00:30.540734 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.540682    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 20:00:30.540734 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.540693    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-209.ec2.internal" event="NodeHasSufficientPID"
Sep 11 20:00:30.563053 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:30.563032    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-209.ec2.internal\" not found" node="ip-10-0-128-209.ec2.internal"
Sep 11 20:00:30.566505 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:30.566480    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-209.ec2.internal\" not found" node="ip-10-0-128-209.ec2.internal"
Sep 11 20:00:30.626400 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:30.626376    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-209.ec2.internal\" not found"
Sep 11 20:00:30.637421 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.637399    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d75655b6d7e7980e905d4bbfadd53b9f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-209.ec2.internal\" (UID: \"d75655b6d7e7980e905d4bbfadd53b9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-209.ec2.internal"
Sep 11 20:00:30.637515 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.637427    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/abeccc9e8a8aea0630d3515e20a0e542-config\") pod \"kube-apiserver-proxy-ip-10-0-128-209.ec2.internal\" (UID: \"abeccc9e8a8aea0630d3515e20a0e542\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-209.ec2.internal"
Sep 11 20:00:30.637515 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.637452    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d75655b6d7e7980e905d4bbfadd53b9f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-209.ec2.internal\" (UID: \"d75655b6d7e7980e905d4bbfadd53b9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-209.ec2.internal"
Sep 11 20:00:30.726640 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:30.726612    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-209.ec2.internal\" not found"
Sep 11 20:00:30.738011 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.737987    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d75655b6d7e7980e905d4bbfadd53b9f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-209.ec2.internal\" (UID: \"d75655b6d7e7980e905d4bbfadd53b9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-209.ec2.internal"
Sep 11 20:00:30.738075 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.738016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d75655b6d7e7980e905d4bbfadd53b9f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-209.ec2.internal\" (UID: \"d75655b6d7e7980e905d4bbfadd53b9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-209.ec2.internal"
Sep 11 20:00:30.738075 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.738032    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/abeccc9e8a8aea0630d3515e20a0e542-config\") pod \"kube-apiserver-proxy-ip-10-0-128-209.ec2.internal\" (UID: \"abeccc9e8a8aea0630d3515e20a0e542\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-209.ec2.internal"
Sep 11 20:00:30.738139 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.738086    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/abeccc9e8a8aea0630d3515e20a0e542-config\") pod \"kube-apiserver-proxy-ip-10-0-128-209.ec2.internal\" (UID: \"abeccc9e8a8aea0630d3515e20a0e542\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-209.ec2.internal"
Sep 11 20:00:30.738139 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.738095    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d75655b6d7e7980e905d4bbfadd53b9f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-209.ec2.internal\" (UID: \"d75655b6d7e7980e905d4bbfadd53b9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-209.ec2.internal"
Sep 11 20:00:30.738139 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.738095    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d75655b6d7e7980e905d4bbfadd53b9f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-209.ec2.internal\" (UID: \"d75655b6d7e7980e905d4bbfadd53b9f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-209.ec2.internal"
Sep 11 20:00:30.827455 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:30.827372    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-209.ec2.internal\" not found"
Sep 11 20:00:30.866939 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.866905    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-209.ec2.internal"
Sep 11 20:00:30.870322 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:30.870300    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-209.ec2.internal"
Sep 11 20:00:30.928222 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:30.928190    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-209.ec2.internal\" not found"
Sep 11 20:00:31.028677 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:31.028651    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-209.ec2.internal\" not found"
Sep 11 20:00:31.129181 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:31.129108    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-209.ec2.internal\" not found"
Sep 11 20:00:31.229741 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:31.229701    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-209.ec2.internal\" not found"
Sep 11 20:00:31.271449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:31.271405    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 20:00:31.272028 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:31.271590    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 20:00:31.272028 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:31.271629    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 20:00:31.328030 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:31.327976    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 19:55:30 +0000 UTC" deadline="2028-06-26 17:24:24.813979974 +0000 UTC"
Sep 11 20:00:31.328030 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:31.328022    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15693h23m53.485964098s"
Sep 11 20:00:31.330802 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:31.330783    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-209.ec2.internal\" not found"
Sep 11 20:00:31.338171 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:31.338151    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 20:00:31.345111 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:31.345078    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75655b6d7e7980e905d4bbfadd53b9f.slice/crio-a580ca519cf77fe205a454434fdd65ca504efda8aeace87428845ece927955bf WatchSource:0}: Error finding container a580ca519cf77fe205a454434fdd65ca504efda8aeace87428845ece927955bf: Status 404 returned error can't find the container with id a580ca519cf77fe205a454434fdd65ca504efda8aeace87428845ece927955bf
Sep 11 20:00:31.351384 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:31.351368    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 20:00:31.364724 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:31.364706    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 20:00:31.435863 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:31.435836    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-209.ec2.internal"
Sep 11 20:00:31.437974 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:31.437933    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-209.ec2.internal" event={"ID":"d75655b6d7e7980e905d4bbfadd53b9f","Type":"ContainerStarted","Data":"a580ca519cf77fe205a454434fdd65ca504efda8aeace87428845ece927955bf"}
Sep 11 20:00:31.447350 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:31.447334    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 20:00:31.448299 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:31.448286    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-209.ec2.internal"
Sep 11 20:00:31.458931 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:31.458916    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 20:00:31.539582 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:31.539547    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabeccc9e8a8aea0630d3515e20a0e542.slice/crio-97ffae0abef905f170512f2ee708a37c551a1eeeb46f2ae9dc8ae6559d623fdb WatchSource:0}: Error finding container 97ffae0abef905f170512f2ee708a37c551a1eeeb46f2ae9dc8ae6559d623fdb: Status 404 returned error can't find the container with id 97ffae0abef905f170512f2ee708a37c551a1eeeb46f2ae9dc8ae6559d623fdb
Sep 11 20:00:32.313016 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.312973    2580 apiserver.go:52] "Watching apiserver"
Sep 11 20:00:32.320615 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.320577    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 20:00:32.322819 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.322796    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 20:00:32.323281 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.323243    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-j9pns","openshift-image-registry/node-ca-knjv8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-209.ec2.internal","openshift-multus/multus-8nw2z","openshift-network-diagnostics/network-check-target-mrmpn","kube-system/global-pull-secret-syncer-95wcq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss","openshift-multus/multus-additional-cni-plugins-9n8ls","openshift-multus/network-metrics-daemon-fp6qw","openshift-network-operator/iptables-alerter-88pnr","openshift-ovn-kubernetes/ovnkube-node-j7pjf","kube-system/konnectivity-agent-vg4js","kube-system/kube-apiserver-proxy-ip-10-0-128-209.ec2.internal","openshift-cluster-node-tuning-operator/tuned-bpst7"]
Sep 11 20:00:32.325133 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.325110    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss"
Sep 11 20:00:32.326588 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.326562    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.327671 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.327649    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 20:00:32.327840 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.327824    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.328246 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.328033    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 20:00:32.328246 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.328056    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 20:00:32.328422 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.328397    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z4fwg\""
Sep 11 20:00:32.329075 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.329056    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 20:00:32.329587 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.329370    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 20:00:32.329587 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.329447    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 20:00:32.329756 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.329654    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.330095 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.330072    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5wxb6\""
Sep 11 20:00:32.330191 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.330149    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 20:00:32.330253 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.330218    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 20:00:32.330315 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.330277    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 20:00:32.331188 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.331146    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:00:32.331288 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:32.331232    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrmpn" podUID="0a30ff2a-a013-4153-8a18-7a8aa6ec98f0"
Sep 11 20:00:32.331903 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.331562    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 20:00:32.331903 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.331606    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 20:00:32.331903 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.331702    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7zncg\""
Sep 11 20:00:32.332594 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.332570    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f6rpl\""
Sep 11 20:00:32.332594 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.332586    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 20:00:32.332754 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.332664    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 20:00:32.333292 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.332870    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 20:00:32.333292 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.332926    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 20:00:32.333292 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.333142    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:32.333292 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:32.333203    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95wcq" podUID="2936adde-ce1b-4859-858d-db3584a92609"
Sep 11 20:00:32.334673 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.334655    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:32.334773 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:32.334743    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp6qw" podUID="dda12783-5f3e-4a3e-9fde-4681797a7b60"
Sep 11 20:00:32.336287 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.336258    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 20:00:32.336452 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.336433    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9n8ls"
Sep 11 20:00:32.338282 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.338039    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-knjv8"
Sep 11 20:00:32.339658 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.339634    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 20:00:32.339753 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.339677    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5b95w\""
Sep 11 20:00:32.340043 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.340023    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 20:00:32.340243 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.340222    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-88pnr"
Sep 11 20:00:32.341840 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.341456    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 20:00:32.341840 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.341530    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 20:00:32.342598 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.342020    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jdjsc\""
Sep 11 20:00:32.342598 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.342328    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 20:00:32.342766 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.342756    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j9pns"
Sep 11 20:00:32.342941 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.342920    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 20:00:32.343249 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.343229    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tmjvz\""
Sep 11 20:00:32.343312 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.343294    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 20:00:32.343532 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.343236    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 20:00:32.344628 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.344612    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vg4js"
Sep 11 20:00:32.345815 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.345637    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tsbfl\""
Sep 11 20:00:32.345815 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.345704    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 20:00:32.346095 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.345860    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 20:00:32.347174 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda12783-5f3e-4a3e-9fde-4681797a7b60-metrics-certs\") pod \"network-metrics-daemon-fp6qw\" (UID: \"dda12783-5f3e-4a3e-9fde-4681797a7b60\") " pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:32.347269 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347187    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-run-systemd\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.347269 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347213    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-etc-openvswitch\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.347269 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347236    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ws85\" (UniqueName: \"kubernetes.io/projected/562cad6a-184c-4433-be65-0cd67d1a29d2-kube-api-access-2ws85\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.347269 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347264    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2936adde-ce1b-4859-858d-db3584a92609-kubelet-config\") pod \"global-pull-secret-syncer-95wcq\" (UID: \"2936adde-ce1b-4859-858d-db3584a92609\") " pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:32.347463 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347282    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-etc-sysconfig\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.347463 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347297    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-etc-sysctl-d\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.347463 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347311    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-var-lib-kubelet\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.347463 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347338    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-system-cni-dir\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.347463 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347362    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v2nl\" (UniqueName: \"kubernetes.io/projected/0a30ff2a-a013-4153-8a18-7a8aa6ec98f0-kube-api-access-9v2nl\") pod \"network-check-target-mrmpn\" (UID: \"0a30ff2a-a013-4153-8a18-7a8aa6ec98f0\") " pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:00:32.347463 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347446    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9c3797d3-8b51-4fad-9bc8-54ac5e275a02-socket-dir\") pod \"aws-ebs-csi-driver-node-s9gss\" (UID: \"9c3797d3-8b51-4fad-9bc8-54ac5e275a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss"
Sep 11 20:00:32.347683 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347481    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9c3797d3-8b51-4fad-9bc8-54ac5e275a02-etc-selinux\") pod \"aws-ebs-csi-driver-node-s9gss\" (UID: \"9c3797d3-8b51-4fad-9bc8-54ac5e275a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss"
Sep 11 20:00:32.347683 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347509    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-host-cni-netd\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.347683 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347530    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4ceeee54-3eb9-4248-87ff-63b7da3896ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9n8ls\" (UID: \"4ceeee54-3eb9-4248-87ff-63b7da3896ce\") " pod="openshift-multus/multus-additional-cni-plugins-9n8ls"
Sep 11 20:00:32.347683 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347591    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9c3797d3-8b51-4fad-9bc8-54ac5e275a02-registration-dir\") pod \"aws-ebs-csi-driver-node-s9gss\" (UID: \"9c3797d3-8b51-4fad-9bc8-54ac5e275a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss"
Sep 11 20:00:32.347683 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-var-lib-openvswitch\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.347683 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347649    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-etc-modprobe-d\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.347683 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347676    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-host\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.348023 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/562cad6a-184c-4433-be65-0cd67d1a29d2-multus-daemon-config\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.348023 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347719    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-host-run-multus-certs\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.348023 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347761    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4ceeee54-3eb9-4248-87ff-63b7da3896ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9n8ls\" (UID: \"4ceeee54-3eb9-4248-87ff-63b7da3896ce\") " pod="openshift-multus/multus-additional-cni-plugins-9n8ls"
Sep 11 20:00:32.348023 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347798    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-host-kubelet\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.348023 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347823    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a90758be-c0bd-42cd-b5b8-65734b2b810e-tmp\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.348023 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347860    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2936adde-ce1b-4859-858d-db3584a92609-original-pull-secret\") pod \"global-pull-secret-syncer-95wcq\" (UID: \"2936adde-ce1b-4859-858d-db3584a92609\") " pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:32.348023 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347886    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ceeee54-3eb9-4248-87ff-63b7da3896ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9n8ls\" (UID: \"4ceeee54-3eb9-4248-87ff-63b7da3896ce\") " pod="openshift-multus/multus-additional-cni-plugins-9n8ls"
Sep 11 20:00:32.348023 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347919    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-ovn-node-metrics-cert\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.348023 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-host-var-lib-kubelet\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.348023 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.347968    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-multus-conf-dir\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.348023 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348012    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9c3797d3-8b51-4fad-9bc8-54ac5e275a02-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s9gss\" (UID: \"9c3797d3-8b51-4fad-9bc8-54ac5e275a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss"
Sep 11 20:00:32.349029 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-systemd-units\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.349029 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348060    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-multus-socket-dir-parent\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.349029 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348064    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 20:00:32.349029 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348093    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-etc-kubernetes\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.349029 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348098    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xrg7n\""
Sep 11 20:00:32.349029 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348141    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-env-overrides\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.349029 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348200    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-cnibin\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.349029 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348250    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ceeee54-3eb9-4248-87ff-63b7da3896ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-9n8ls\" (UID: \"4ceeee54-3eb9-4248-87ff-63b7da3896ce\") " pod="openshift-multus/multus-additional-cni-plugins-9n8ls"
Sep 11 20:00:32.349029 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348302    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-node-log\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.349029 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348327    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-host-cni-bin\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.349029 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348359    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-etc-kubernetes\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.349029 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348397    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/562cad6a-184c-4433-be65-0cd67d1a29d2-cni-binary-copy\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.349029 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348442    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-host-run-netns\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.349029 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348457    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 20:00:32.349029 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348485    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6crv6\" (UniqueName: \"kubernetes.io/projected/4ceeee54-3eb9-4248-87ff-63b7da3896ce-kube-api-access-6crv6\") pod \"multus-additional-cni-plugins-9n8ls\" (UID: \"4ceeee54-3eb9-4248-87ff-63b7da3896ce\") " pod="openshift-multus/multus-additional-cni-plugins-9n8ls"
Sep 11 20:00:32.349029 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348543    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd8xs\" (UniqueName: \"kubernetes.io/projected/9c3797d3-8b51-4fad-9bc8-54ac5e275a02-kube-api-access-fd8xs\") pod \"aws-ebs-csi-driver-node-s9gss\" (UID: \"9c3797d3-8b51-4fad-9bc8-54ac5e275a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss"
Sep 11 20:00:32.349029 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348573    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-log-socket\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.349029 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-etc-sysctl-conf\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.350248 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ceeee54-3eb9-4248-87ff-63b7da3896ce-cnibin\") pod \"multus-additional-cni-plugins-9n8ls\" (UID: \"4ceeee54-3eb9-4248-87ff-63b7da3896ce\") " pod="openshift-multus/multus-additional-cni-plugins-9n8ls"
Sep 11 20:00:32.350248 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348707    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-ovnkube-script-lib\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.350248 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348739    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ceeee54-3eb9-4248-87ff-63b7da3896ce-system-cni-dir\") pod \"multus-additional-cni-plugins-9n8ls\" (UID: \"4ceeee54-3eb9-4248-87ff-63b7da3896ce\") " pod="openshift-multus/multus-additional-cni-plugins-9n8ls"
Sep 11 20:00:32.350248 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348763    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9c3797d3-8b51-4fad-9bc8-54ac5e275a02-device-dir\") pod \"aws-ebs-csi-driver-node-s9gss\" (UID: \"9c3797d3-8b51-4fad-9bc8-54ac5e275a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss"
Sep 11 20:00:32.350248 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348792    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.350248 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcjm6\" (UniqueName: \"kubernetes.io/projected/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-kube-api-access-zcjm6\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.350248 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348862    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-sys\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.350248 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348886    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-run-openvswitch\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.350248 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-run-ovn\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.350248 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.350248 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.348978    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-etc-systemd\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.350248 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.349008    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-run\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.350248 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.349038    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-multus-cni-dir\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.350248 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.349063    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-host-run-k8s-cni-cncf-io\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.350248 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.349089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-host-var-lib-cni-multus\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.350248 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.349126    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-hostroot\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.351197 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.349154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2936adde-ce1b-4859-858d-db3584a92609-dbus\") pod \"global-pull-secret-syncer-95wcq\" (UID: \"2936adde-ce1b-4859-858d-db3584a92609\") " pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:32.351197 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.349179    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-ovnkube-config\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.351197 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.349204    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvhl9\" (UniqueName: \"kubernetes.io/projected/dda12783-5f3e-4a3e-9fde-4681797a7b60-kube-api-access-mvhl9\") pod \"network-metrics-daemon-fp6qw\" (UID: \"dda12783-5f3e-4a3e-9fde-4681797a7b60\") " pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:32.351197 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.349228    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-host-run-netns\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.351197 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.349250    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a90758be-c0bd-42cd-b5b8-65734b2b810e-etc-tuned\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.351197 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.349275    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9c3797d3-8b51-4fad-9bc8-54ac5e275a02-sys-fs\") pod \"aws-ebs-csi-driver-node-s9gss\" (UID: \"9c3797d3-8b51-4fad-9bc8-54ac5e275a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss"
Sep 11 20:00:32.351197 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.349301    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-host-slash\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.351197 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.349326    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-lib-modules\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.351197 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.349351    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgv7x\" (UniqueName: \"kubernetes.io/projected/a90758be-c0bd-42cd-b5b8-65734b2b810e-kube-api-access-lgv7x\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.351197 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.349375    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-host-var-lib-cni-bin\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.353694 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.353539    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 20:00:32.420404 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.420367    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fnzr8"
Sep 11 20:00:32.429729 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.429443    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fnzr8"
Sep 11 20:00:32.437201 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.437171    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 20:00:32.440609 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.440531    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-209.ec2.internal" event={"ID":"abeccc9e8a8aea0630d3515e20a0e542","Type":"ContainerStarted","Data":"97ffae0abef905f170512f2ee708a37c551a1eeeb46f2ae9dc8ae6559d623fdb"}
Sep 11 20:00:32.450076 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450029    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9c3797d3-8b51-4fad-9bc8-54ac5e275a02-registration-dir\") pod \"aws-ebs-csi-driver-node-s9gss\" (UID: \"9c3797d3-8b51-4fad-9bc8-54ac5e275a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss"
Sep 11 20:00:32.450200 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450094    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-var-lib-openvswitch\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.450200 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-etc-modprobe-d\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.450200 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450130    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9c3797d3-8b51-4fad-9bc8-54ac5e275a02-registration-dir\") pod \"aws-ebs-csi-driver-node-s9gss\" (UID: \"9c3797d3-8b51-4fad-9bc8-54ac5e275a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss"
Sep 11 20:00:32.450200 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-host\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.450200 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450174    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/562cad6a-184c-4433-be65-0cd67d1a29d2-multus-daemon-config\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.450200 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450192    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-var-lib-openvswitch\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.450544 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450199    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-host-run-multus-certs\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.450544 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450233    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4ceeee54-3eb9-4248-87ff-63b7da3896ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9n8ls\" (UID: \"4ceeee54-3eb9-4248-87ff-63b7da3896ce\") " pod="openshift-multus/multus-additional-cni-plugins-9n8ls"
Sep 11 20:00:32.450544 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450241    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-host\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.450544 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450260    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-host-kubelet\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.450544 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450284    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a90758be-c0bd-42cd-b5b8-65734b2b810e-tmp\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.450544 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450309    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2936adde-ce1b-4859-858d-db3584a92609-original-pull-secret\") pod \"global-pull-secret-syncer-95wcq\" (UID: \"2936adde-ce1b-4859-858d-db3584a92609\") " pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:32.450544 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450311    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-etc-modprobe-d\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.450544 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450332    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ceeee54-3eb9-4248-87ff-63b7da3896ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9n8ls\" (UID: \"4ceeee54-3eb9-4248-87ff-63b7da3896ce\") " pod="openshift-multus/multus-additional-cni-plugins-9n8ls"
Sep 11 20:00:32.450544 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450372    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f665fa9b-d485-4c67-8979-b4cf0c365dbb-iptables-alerter-script\") pod \"iptables-alerter-88pnr\" (UID: \"f665fa9b-d485-4c67-8979-b4cf0c365dbb\") " pod="openshift-network-operator/iptables-alerter-88pnr"
Sep 11 20:00:32.450544 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-ovn-node-metrics-cert\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.450544 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450437    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-host-var-lib-kubelet\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.450544 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-multus-conf-dir\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.450544 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4ceeee54-3eb9-4248-87ff-63b7da3896ce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9n8ls\" (UID: \"4ceeee54-3eb9-4248-87ff-63b7da3896ce\") " pod="openshift-multus/multus-additional-cni-plugins-9n8ls"
Sep 11 20:00:32.450544 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450491    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9c3797d3-8b51-4fad-9bc8-54ac5e275a02-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s9gss\" (UID: \"9c3797d3-8b51-4fad-9bc8-54ac5e275a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss"
Sep 11 20:00:32.450544 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450531    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-systemd-units\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.451268 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450560    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9c3797d3-8b51-4fad-9bc8-54ac5e275a02-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s9gss\" (UID: \"9c3797d3-8b51-4fad-9bc8-54ac5e275a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss"
Sep 11 20:00:32.451268 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450575    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-multus-socket-dir-parent\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.451268 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450603    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-etc-kubernetes\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.451268 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-env-overrides\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.451268 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450673    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-cnibin\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.451268 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ceeee54-3eb9-4248-87ff-63b7da3896ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-9n8ls\" (UID: \"4ceeee54-3eb9-4248-87ff-63b7da3896ce\") " pod="openshift-multus/multus-additional-cni-plugins-9n8ls"
Sep 11 20:00:32.451268 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450725    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0ddb4097-dd15-4361-8302-fded613f0568-konnectivity-ca\") pod \"konnectivity-agent-vg4js\" (UID: \"0ddb4097-dd15-4361-8302-fded613f0568\") " pod="kube-system/konnectivity-agent-vg4js"
Sep 11 20:00:32.451268 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450753    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-node-log\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.451268 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-host-cni-bin\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.451268 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-etc-kubernetes\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.451268 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450834    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/562cad6a-184c-4433-be65-0cd67d1a29d2-cni-binary-copy\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.451268 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450859    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-host-run-netns\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.451268 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450910    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6crv6\" (UniqueName: \"kubernetes.io/projected/4ceeee54-3eb9-4248-87ff-63b7da3896ce-kube-api-access-6crv6\") pod \"multus-additional-cni-plugins-9n8ls\" (UID: \"4ceeee54-3eb9-4248-87ff-63b7da3896ce\") " pod="openshift-multus/multus-additional-cni-plugins-9n8ls"
Sep 11 20:00:32.451268 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450922    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/562cad6a-184c-4433-be65-0cd67d1a29d2-multus-daemon-config\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.451268 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450938    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0ddb4097-dd15-4361-8302-fded613f0568-agent-certs\") pod \"konnectivity-agent-vg4js\" (UID: \"0ddb4097-dd15-4361-8302-fded613f0568\") " pod="kube-system/konnectivity-agent-vg4js"
Sep 11 20:00:32.451268 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450971    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd8xs\" (UniqueName: \"kubernetes.io/projected/9c3797d3-8b51-4fad-9bc8-54ac5e275a02-kube-api-access-fd8xs\") pod \"aws-ebs-csi-driver-node-s9gss\" (UID: \"9c3797d3-8b51-4fad-9bc8-54ac5e275a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss"
Sep 11 20:00:32.451268 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450997    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-log-socket\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.452168 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451015    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-host-var-lib-kubelet\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.452168 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-etc-sysctl-conf\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.452168 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ceeee54-3eb9-4248-87ff-63b7da3896ce-cnibin\") pod \"multus-additional-cni-plugins-9n8ls\" (UID: \"4ceeee54-3eb9-4248-87ff-63b7da3896ce\") " pod="openshift-multus/multus-additional-cni-plugins-9n8ls"
Sep 11 20:00:32.452168 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-multus-socket-dir-parent\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.452168 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.450284    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-host-run-multus-certs\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.452168 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451092    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 20:00:32.452168 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451126    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4ceeee54-3eb9-4248-87ff-63b7da3896ce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9n8ls\" (UID: \"4ceeee54-3eb9-4248-87ff-63b7da3896ce\") " pod="openshift-multus/multus-additional-cni-plugins-9n8ls"
Sep 11 20:00:32.452168 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451174    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-etc-kubernetes\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.452168 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451184    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-host-run-netns\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.452168 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451231    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-etc-sysctl-conf\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.452168 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:32.451283    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 20:00:32.452168 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:32.451350    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2936adde-ce1b-4859-858d-db3584a92609-original-pull-secret podName:2936adde-ce1b-4859-858d-db3584a92609 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:32.951316758 +0000 UTC m=+2.943479008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2936adde-ce1b-4859-858d-db3584a92609-original-pull-secret") pod "global-pull-secret-syncer-95wcq" (UID: "2936adde-ce1b-4859-858d-db3584a92609") : object "kube-system"/"original-pull-secret" not registered
Sep 11 20:00:32.452168 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451524    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-log-socket\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.452168 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451548    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4ceeee54-3eb9-4248-87ff-63b7da3896ce-cni-binary-copy\") pod \"multus-additional-cni-plugins-9n8ls\" (UID: \"4ceeee54-3eb9-4248-87ff-63b7da3896ce\") " pod="openshift-multus/multus-additional-cni-plugins-9n8ls"
Sep 11 20:00:32.452168 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451589    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-systemd-units\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.452168 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451616    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4ceeee54-3eb9-4248-87ff-63b7da3896ce-cnibin\") pod \"multus-additional-cni-plugins-9n8ls\" (UID: \"4ceeee54-3eb9-4248-87ff-63b7da3896ce\") " pod="openshift-multus/multus-additional-cni-plugins-9n8ls"
Sep 11 20:00:32.452168 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451634    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-host-kubelet\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.452168 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451672    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-node-log\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.453156 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451715    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/562cad6a-184c-4433-be65-0cd67d1a29d2-cni-binary-copy\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.453156 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451721    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-host-cni-bin\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.453156 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-ovnkube-script-lib\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.453156 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451763    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-cnibin\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.453156 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451772    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-multus-conf-dir\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.453156 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ceeee54-3eb9-4248-87ff-63b7da3896ce-system-cni-dir\") pod \"multus-additional-cni-plugins-9n8ls\" (UID: \"4ceeee54-3eb9-4248-87ff-63b7da3896ce\") " pod="openshift-multus/multus-additional-cni-plugins-9n8ls"
Sep 11 20:00:32.453156 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451849    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-etc-kubernetes\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.453156 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9c3797d3-8b51-4fad-9bc8-54ac5e275a02-device-dir\") pod \"aws-ebs-csi-driver-node-s9gss\" (UID: \"9c3797d3-8b51-4fad-9bc8-54ac5e275a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss"
Sep 11 20:00:32.453156 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451931    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.453156 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451942    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9c3797d3-8b51-4fad-9bc8-54ac5e275a02-device-dir\") pod \"aws-ebs-csi-driver-node-s9gss\" (UID: \"9c3797d3-8b51-4fad-9bc8-54ac5e275a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss"
Sep 11 20:00:32.453156 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451978    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcjm6\" (UniqueName: \"kubernetes.io/projected/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-kube-api-access-zcjm6\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.453156 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.451984    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.453156 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452026    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-sys\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.453156 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452027    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4ceeee54-3eb9-4248-87ff-63b7da3896ce-system-cni-dir\") pod \"multus-additional-cni-plugins-9n8ls\" (UID: \"4ceeee54-3eb9-4248-87ff-63b7da3896ce\") " pod="openshift-multus/multus-additional-cni-plugins-9n8ls"
Sep 11 20:00:32.453156 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-env-overrides\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.453156 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452170    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83199f92-0393-4012-96cb-bb5d3172786f-tmp-dir\") pod \"node-resolver-j9pns\" (UID: \"83199f92-0393-4012-96cb-bb5d3172786f\") " pod="openshift-dns/node-resolver-j9pns"
Sep 11 20:00:32.453156 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452198    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-run-openvswitch\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.454084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452232    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-run-ovn\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.454084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452260    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.454084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452267    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-sys\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.454084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452284    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-run-openvswitch\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.454084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-etc-systemd\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.454084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452360    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-run\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.454084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452385    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.454084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452426    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-multus-cni-dir\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.454084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452438    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-etc-systemd\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.454084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-host-run-k8s-cni-cncf-io\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.454084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452482    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-run-ovn\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.454084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-run\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.454084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452615    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-multus-cni-dir\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.454084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452615    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-host-var-lib-cni-multus\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.454084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452654    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-host-var-lib-cni-multus\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.454084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-hostroot\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.454084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2936adde-ce1b-4859-858d-db3584a92609-dbus\") pod \"global-pull-secret-syncer-95wcq\" (UID: \"2936adde-ce1b-4859-858d-db3584a92609\") " pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:32.454084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452751    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/83199f92-0393-4012-96cb-bb5d3172786f-hosts-file\") pod \"node-resolver-j9pns\" (UID: \"83199f92-0393-4012-96cb-bb5d3172786f\") " pod="openshift-dns/node-resolver-j9pns"
Sep 11 20:00:32.455047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452312    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-ovnkube-script-lib\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.455047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452793    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-host-run-k8s-cni-cncf-io\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.455047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-ovnkube-config\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.455047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452869    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2936adde-ce1b-4859-858d-db3584a92609-dbus\") pod \"global-pull-secret-syncer-95wcq\" (UID: \"2936adde-ce1b-4859-858d-db3584a92609\") " pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:32.455047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvhl9\" (UniqueName: \"kubernetes.io/projected/dda12783-5f3e-4a3e-9fde-4681797a7b60-kube-api-access-mvhl9\") pod \"network-metrics-daemon-fp6qw\" (UID: \"dda12783-5f3e-4a3e-9fde-4681797a7b60\") " pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:32.455047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452885    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-hostroot\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.455047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-host-run-netns\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.455047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a90758be-c0bd-42cd-b5b8-65734b2b810e-etc-tuned\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.455047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.452985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9c3797d3-8b51-4fad-9bc8-54ac5e275a02-sys-fs\") pod \"aws-ebs-csi-driver-node-s9gss\" (UID: \"9c3797d3-8b51-4fad-9bc8-54ac5e275a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss"
Sep 11 20:00:32.455047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-host-slash\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.455047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453054    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-lib-modules\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.455047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453081    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgv7x\" (UniqueName: \"kubernetes.io/projected/a90758be-c0bd-42cd-b5b8-65734b2b810e-kube-api-access-lgv7x\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.455047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453106    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-host-var-lib-cni-bin\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.455047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453134    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda12783-5f3e-4a3e-9fde-4681797a7b60-metrics-certs\") pod \"network-metrics-daemon-fp6qw\" (UID: \"dda12783-5f3e-4a3e-9fde-4681797a7b60\") " pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:32.455047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453185    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-run-systemd\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.455047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453213    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-etc-openvswitch\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.455047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ws85\" (UniqueName: \"kubernetes.io/projected/562cad6a-184c-4433-be65-0cd67d1a29d2-kube-api-access-2ws85\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.455911 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453271    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2936adde-ce1b-4859-858d-db3584a92609-kubelet-config\") pod \"global-pull-secret-syncer-95wcq\" (UID: \"2936adde-ce1b-4859-858d-db3584a92609\") " pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:32.455911 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-lib-modules\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.455911 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453298    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59344c06-c093-43c1-8ef5-9e3df17dd1cd-host\") pod \"node-ca-knjv8\" (UID: \"59344c06-c093-43c1-8ef5-9e3df17dd1cd\") " pod="openshift-image-registry/node-ca-knjv8"
Sep 11 20:00:32.455911 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp948\" (UniqueName: \"kubernetes.io/projected/f665fa9b-d485-4c67-8979-b4cf0c365dbb-kube-api-access-mp948\") pod \"iptables-alerter-88pnr\" (UID: \"f665fa9b-d485-4c67-8979-b4cf0c365dbb\") " pod="openshift-network-operator/iptables-alerter-88pnr"
Sep 11 20:00:32.455911 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:32.453361    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 20:00:32.455911 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-etc-sysconfig\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.455911 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453424    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-etc-sysctl-d\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.455911 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453450    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-var-lib-kubelet\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.455911 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-system-cni-dir\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.455911 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v2nl\" (UniqueName: \"kubernetes.io/projected/0a30ff2a-a013-4153-8a18-7a8aa6ec98f0-kube-api-access-9v2nl\") pod \"network-check-target-mrmpn\" (UID: \"0a30ff2a-a013-4153-8a18-7a8aa6ec98f0\") " pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:00:32.455911 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453493    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2936adde-ce1b-4859-858d-db3584a92609-kubelet-config\") pod \"global-pull-secret-syncer-95wcq\" (UID: \"2936adde-ce1b-4859-858d-db3584a92609\") " pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:32.455911 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/59344c06-c093-43c1-8ef5-9e3df17dd1cd-serviceca\") pod \"node-ca-knjv8\" (UID: \"59344c06-c093-43c1-8ef5-9e3df17dd1cd\") " pod="openshift-image-registry/node-ca-knjv8"
Sep 11 20:00:32.455911 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453576    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x2h2\" (UniqueName: \"kubernetes.io/projected/59344c06-c093-43c1-8ef5-9e3df17dd1cd-kube-api-access-8x2h2\") pod \"node-ca-knjv8\" (UID: \"59344c06-c093-43c1-8ef5-9e3df17dd1cd\") " pod="openshift-image-registry/node-ca-knjv8"
Sep 11 20:00:32.455911 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453590    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-ovnkube-config\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.455911 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453606    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f665fa9b-d485-4c67-8979-b4cf0c365dbb-host-slash\") pod \"iptables-alerter-88pnr\" (UID: \"f665fa9b-d485-4c67-8979-b4cf0c365dbb\") " pod="openshift-network-operator/iptables-alerter-88pnr"
Sep 11 20:00:32.455911 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:32.453633    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda12783-5f3e-4a3e-9fde-4681797a7b60-metrics-certs podName:dda12783-5f3e-4a3e-9fde-4681797a7b60 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:32.953616143 +0000 UTC m=+2.945778403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dda12783-5f3e-4a3e-9fde-4681797a7b60-metrics-certs") pod "network-metrics-daemon-fp6qw" (UID: "dda12783-5f3e-4a3e-9fde-4681797a7b60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 20:00:32.455911 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453671    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-etc-sysconfig\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.456793 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453674    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9c3797d3-8b51-4fad-9bc8-54ac5e275a02-socket-dir\") pod \"aws-ebs-csi-driver-node-s9gss\" (UID: \"9c3797d3-8b51-4fad-9bc8-54ac5e275a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss"
Sep 11 20:00:32.456793 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453694    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-var-lib-kubelet\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.456793 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9c3797d3-8b51-4fad-9bc8-54ac5e275a02-etc-selinux\") pod \"aws-ebs-csi-driver-node-s9gss\" (UID: \"9c3797d3-8b51-4fad-9bc8-54ac5e275a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss"
Sep 11 20:00:32.456793 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453721    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9c3797d3-8b51-4fad-9bc8-54ac5e275a02-sys-fs\") pod \"aws-ebs-csi-driver-node-s9gss\" (UID: \"9c3797d3-8b51-4fad-9bc8-54ac5e275a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss"
Sep 11 20:00:32.456793 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453736    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-host-cni-netd\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.456793 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453767    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-system-cni-dir\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.456793 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453776    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-host-slash\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.456793 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453781    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-host-cni-netd\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.456793 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453810    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-host-run-netns\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.456793 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453816    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-run-systemd\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.456793 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453830    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/562cad6a-184c-4433-be65-0cd67d1a29d2-host-var-lib-cni-bin\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.456793 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453847    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9c3797d3-8b51-4fad-9bc8-54ac5e275a02-socket-dir\") pod \"aws-ebs-csi-driver-node-s9gss\" (UID: \"9c3797d3-8b51-4fad-9bc8-54ac5e275a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss"
Sep 11 20:00:32.456793 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453847    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-etc-openvswitch\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.456793 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453872    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4ceeee54-3eb9-4248-87ff-63b7da3896ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9n8ls\" (UID: \"4ceeee54-3eb9-4248-87ff-63b7da3896ce\") " pod="openshift-multus/multus-additional-cni-plugins-9n8ls"
Sep 11 20:00:32.456793 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453904    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9c3797d3-8b51-4fad-9bc8-54ac5e275a02-etc-selinux\") pod \"aws-ebs-csi-driver-node-s9gss\" (UID: \"9c3797d3-8b51-4fad-9bc8-54ac5e275a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss"
Sep 11 20:00:32.456793 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5swr\" (UniqueName: \"kubernetes.io/projected/83199f92-0393-4012-96cb-bb5d3172786f-kube-api-access-m5swr\") pod \"node-resolver-j9pns\" (UID: \"83199f92-0393-4012-96cb-bb5d3172786f\") " pod="openshift-dns/node-resolver-j9pns"
Sep 11 20:00:32.456793 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.453989    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a90758be-c0bd-42cd-b5b8-65734b2b810e-etc-sysctl-d\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.457641 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.454376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4ceeee54-3eb9-4248-87ff-63b7da3896ce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9n8ls\" (UID: \"4ceeee54-3eb9-4248-87ff-63b7da3896ce\") " pod="openshift-multus/multus-additional-cni-plugins-9n8ls"
Sep 11 20:00:32.457641 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.456268    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a90758be-c0bd-42cd-b5b8-65734b2b810e-tmp\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.457641 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.456351    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-ovn-node-metrics-cert\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.457641 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.456844    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a90758be-c0bd-42cd-b5b8-65734b2b810e-etc-tuned\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.470662 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:32.469912    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 20:00:32.470662 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:32.469943    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 20:00:32.470662 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:32.469959    2580 projected.go:196] Error preparing data for projected volume kube-api-access-9v2nl for pod openshift-network-diagnostics/network-check-target-mrmpn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 20:00:32.470662 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:32.470039    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a30ff2a-a013-4153-8a18-7a8aa6ec98f0-kube-api-access-9v2nl podName:0a30ff2a-a013-4153-8a18-7a8aa6ec98f0 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:32.970019285 +0000 UTC m=+2.962181556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9v2nl" (UniqueName: "kubernetes.io/projected/0a30ff2a-a013-4153-8a18-7a8aa6ec98f0-kube-api-access-9v2nl") pod "network-check-target-mrmpn" (UID: "0a30ff2a-a013-4153-8a18-7a8aa6ec98f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 20:00:32.472881 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.472852    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvhl9\" (UniqueName: \"kubernetes.io/projected/dda12783-5f3e-4a3e-9fde-4681797a7b60-kube-api-access-mvhl9\") pod \"network-metrics-daemon-fp6qw\" (UID: \"dda12783-5f3e-4a3e-9fde-4681797a7b60\") " pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:32.473000 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.472862    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcjm6\" (UniqueName: \"kubernetes.io/projected/5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6-kube-api-access-zcjm6\") pod \"ovnkube-node-j7pjf\" (UID: \"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.473577 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.473479    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ws85\" (UniqueName: \"kubernetes.io/projected/562cad6a-184c-4433-be65-0cd67d1a29d2-kube-api-access-2ws85\") pod \"multus-8nw2z\" (UID: \"562cad6a-184c-4433-be65-0cd67d1a29d2\") " pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.473693 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.473585    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6crv6\" (UniqueName: \"kubernetes.io/projected/4ceeee54-3eb9-4248-87ff-63b7da3896ce-kube-api-access-6crv6\") pod \"multus-additional-cni-plugins-9n8ls\" (UID: \"4ceeee54-3eb9-4248-87ff-63b7da3896ce\") " pod="openshift-multus/multus-additional-cni-plugins-9n8ls"
Sep 11 20:00:32.474222 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.474201    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd8xs\" (UniqueName: \"kubernetes.io/projected/9c3797d3-8b51-4fad-9bc8-54ac5e275a02-kube-api-access-fd8xs\") pod \"aws-ebs-csi-driver-node-s9gss\" (UID: \"9c3797d3-8b51-4fad-9bc8-54ac5e275a02\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss"
Sep 11 20:00:32.475114 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.475090    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgv7x\" (UniqueName: \"kubernetes.io/projected/a90758be-c0bd-42cd-b5b8-65734b2b810e-kube-api-access-lgv7x\") pod \"tuned-bpst7\" (UID: \"a90758be-c0bd-42cd-b5b8-65734b2b810e\") " pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.523975 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.523943    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 20:00:32.554738 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.554699    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0ddb4097-dd15-4361-8302-fded613f0568-konnectivity-ca\") pod \"konnectivity-agent-vg4js\" (UID: \"0ddb4097-dd15-4361-8302-fded613f0568\") " pod="kube-system/konnectivity-agent-vg4js"
Sep 11 20:00:32.554738 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.554750    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0ddb4097-dd15-4361-8302-fded613f0568-agent-certs\") pod \"konnectivity-agent-vg4js\" (UID: \"0ddb4097-dd15-4361-8302-fded613f0568\") " pod="kube-system/konnectivity-agent-vg4js"
Sep 11 20:00:32.554987 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.554873    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83199f92-0393-4012-96cb-bb5d3172786f-tmp-dir\") pod \"node-resolver-j9pns\" (UID: \"83199f92-0393-4012-96cb-bb5d3172786f\") " pod="openshift-dns/node-resolver-j9pns"
Sep 11 20:00:32.554987 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.554912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/83199f92-0393-4012-96cb-bb5d3172786f-hosts-file\") pod \"node-resolver-j9pns\" (UID: \"83199f92-0393-4012-96cb-bb5d3172786f\") " pod="openshift-dns/node-resolver-j9pns"
Sep 11 20:00:32.554987 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.554960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59344c06-c093-43c1-8ef5-9e3df17dd1cd-host\") pod \"node-ca-knjv8\" (UID: \"59344c06-c093-43c1-8ef5-9e3df17dd1cd\") " pod="openshift-image-registry/node-ca-knjv8"
Sep 11 20:00:32.555149 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.554984    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp948\" (UniqueName: \"kubernetes.io/projected/f665fa9b-d485-4c67-8979-b4cf0c365dbb-kube-api-access-mp948\") pod \"iptables-alerter-88pnr\" (UID: \"f665fa9b-d485-4c67-8979-b4cf0c365dbb\") " pod="openshift-network-operator/iptables-alerter-88pnr"
Sep 11 20:00:32.555149 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.555033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/59344c06-c093-43c1-8ef5-9e3df17dd1cd-serviceca\") pod \"node-ca-knjv8\" (UID: \"59344c06-c093-43c1-8ef5-9e3df17dd1cd\") " pod="openshift-image-registry/node-ca-knjv8"
Sep 11 20:00:32.555149 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.555059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x2h2\" (UniqueName: \"kubernetes.io/projected/59344c06-c093-43c1-8ef5-9e3df17dd1cd-kube-api-access-8x2h2\") pod \"node-ca-knjv8\" (UID: \"59344c06-c093-43c1-8ef5-9e3df17dd1cd\") " pod="openshift-image-registry/node-ca-knjv8"
Sep 11 20:00:32.555149 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.555082    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f665fa9b-d485-4c67-8979-b4cf0c365dbb-host-slash\") pod \"iptables-alerter-88pnr\" (UID: \"f665fa9b-d485-4c67-8979-b4cf0c365dbb\") " pod="openshift-network-operator/iptables-alerter-88pnr"
Sep 11 20:00:32.555149 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.555111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5swr\" (UniqueName: \"kubernetes.io/projected/83199f92-0393-4012-96cb-bb5d3172786f-kube-api-access-m5swr\") pod \"node-resolver-j9pns\" (UID: \"83199f92-0393-4012-96cb-bb5d3172786f\") " pod="openshift-dns/node-resolver-j9pns"
Sep 11 20:00:32.555399 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.555159    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f665fa9b-d485-4c67-8979-b4cf0c365dbb-iptables-alerter-script\") pod \"iptables-alerter-88pnr\" (UID: \"f665fa9b-d485-4c67-8979-b4cf0c365dbb\") " pod="openshift-network-operator/iptables-alerter-88pnr"
Sep 11 20:00:32.555399 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.555360    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0ddb4097-dd15-4361-8302-fded613f0568-konnectivity-ca\") pod \"konnectivity-agent-vg4js\" (UID: \"0ddb4097-dd15-4361-8302-fded613f0568\") " pod="kube-system/konnectivity-agent-vg4js"
Sep 11 20:00:32.555507 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.555463    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/83199f92-0393-4012-96cb-bb5d3172786f-hosts-file\") pod \"node-resolver-j9pns\" (UID: \"83199f92-0393-4012-96cb-bb5d3172786f\") " pod="openshift-dns/node-resolver-j9pns"
Sep 11 20:00:32.555507 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.555473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59344c06-c093-43c1-8ef5-9e3df17dd1cd-host\") pod \"node-ca-knjv8\" (UID: \"59344c06-c093-43c1-8ef5-9e3df17dd1cd\") " pod="openshift-image-registry/node-ca-knjv8"
Sep 11 20:00:32.555630 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.555511    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/59344c06-c093-43c1-8ef5-9e3df17dd1cd-serviceca\") pod \"node-ca-knjv8\" (UID: \"59344c06-c093-43c1-8ef5-9e3df17dd1cd\") " pod="openshift-image-registry/node-ca-knjv8"
Sep 11 20:00:32.555837 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.555770    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f665fa9b-d485-4c67-8979-b4cf0c365dbb-host-slash\") pod \"iptables-alerter-88pnr\" (UID: \"f665fa9b-d485-4c67-8979-b4cf0c365dbb\") " pod="openshift-network-operator/iptables-alerter-88pnr"
Sep 11 20:00:32.555837 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.555782    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83199f92-0393-4012-96cb-bb5d3172786f-tmp-dir\") pod \"node-resolver-j9pns\" (UID: \"83199f92-0393-4012-96cb-bb5d3172786f\") " pod="openshift-dns/node-resolver-j9pns"
Sep 11 20:00:32.556062 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.556035    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f665fa9b-d485-4c67-8979-b4cf0c365dbb-iptables-alerter-script\") pod \"iptables-alerter-88pnr\" (UID: \"f665fa9b-d485-4c67-8979-b4cf0c365dbb\") " pod="openshift-network-operator/iptables-alerter-88pnr"
Sep 11 20:00:32.558258 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.558224    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0ddb4097-dd15-4361-8302-fded613f0568-agent-certs\") pod \"konnectivity-agent-vg4js\" (UID: \"0ddb4097-dd15-4361-8302-fded613f0568\") " pod="kube-system/konnectivity-agent-vg4js"
Sep 11 20:00:32.567148 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.567082    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp948\" (UniqueName: \"kubernetes.io/projected/f665fa9b-d485-4c67-8979-b4cf0c365dbb-kube-api-access-mp948\") pod \"iptables-alerter-88pnr\" (UID: \"f665fa9b-d485-4c67-8979-b4cf0c365dbb\") " pod="openshift-network-operator/iptables-alerter-88pnr"
Sep 11 20:00:32.568402 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.568373    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5swr\" (UniqueName: \"kubernetes.io/projected/83199f92-0393-4012-96cb-bb5d3172786f-kube-api-access-m5swr\") pod \"node-resolver-j9pns\" (UID: \"83199f92-0393-4012-96cb-bb5d3172786f\") " pod="openshift-dns/node-resolver-j9pns"
Sep 11 20:00:32.568801 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.568781    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x2h2\" (UniqueName: \"kubernetes.io/projected/59344c06-c093-43c1-8ef5-9e3df17dd1cd-kube-api-access-8x2h2\") pod \"node-ca-knjv8\" (UID: \"59344c06-c093-43c1-8ef5-9e3df17dd1cd\") " pod="openshift-image-registry/node-ca-knjv8"
Sep 11 20:00:32.638205 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.638169    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss"
Sep 11 20:00:32.649515 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.649471    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:32.663686 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.663664    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bpst7"
Sep 11 20:00:32.669771 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.669749    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8nw2z"
Sep 11 20:00:32.677516 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.677483    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9n8ls"
Sep 11 20:00:32.684376 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.684354    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-knjv8"
Sep 11 20:00:32.694258 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.694242    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-88pnr"
Sep 11 20:00:32.702081 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.702060    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j9pns"
Sep 11 20:00:32.708993 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.708971    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vg4js"
Sep 11 20:00:32.957938 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.957897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda12783-5f3e-4a3e-9fde-4681797a7b60-metrics-certs\") pod \"network-metrics-daemon-fp6qw\" (UID: \"dda12783-5f3e-4a3e-9fde-4681797a7b60\") " pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:32.958154 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:32.957977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2936adde-ce1b-4859-858d-db3584a92609-original-pull-secret\") pod \"global-pull-secret-syncer-95wcq\" (UID: \"2936adde-ce1b-4859-858d-db3584a92609\") " pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:32.958154 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:32.958084    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 20:00:32.958154 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:32.958153    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda12783-5f3e-4a3e-9fde-4681797a7b60-metrics-certs podName:dda12783-5f3e-4a3e-9fde-4681797a7b60 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:33.958136561 +0000 UTC m=+3.950298810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dda12783-5f3e-4a3e-9fde-4681797a7b60-metrics-certs") pod "network-metrics-daemon-fp6qw" (UID: "dda12783-5f3e-4a3e-9fde-4681797a7b60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 20:00:32.958353 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:32.958085    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 20:00:32.958353 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:32.958231    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2936adde-ce1b-4859-858d-db3584a92609-original-pull-secret podName:2936adde-ce1b-4859-858d-db3584a92609 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:33.958217354 +0000 UTC m=+3.950379605 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2936adde-ce1b-4859-858d-db3584a92609-original-pull-secret") pod "global-pull-secret-syncer-95wcq" (UID: "2936adde-ce1b-4859-858d-db3584a92609") : object "kube-system"/"original-pull-secret" not registered
Sep 11 20:00:33.059206 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:33.059169    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v2nl\" (UniqueName: \"kubernetes.io/projected/0a30ff2a-a013-4153-8a18-7a8aa6ec98f0-kube-api-access-9v2nl\") pod \"network-check-target-mrmpn\" (UID: \"0a30ff2a-a013-4153-8a18-7a8aa6ec98f0\") " pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:00:33.059413 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:33.059365    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 20:00:33.059413 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:33.059389    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 20:00:33.059413 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:33.059399    2580 projected.go:196] Error preparing data for projected volume kube-api-access-9v2nl for pod openshift-network-diagnostics/network-check-target-mrmpn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 20:00:33.059610 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:33.059467    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a30ff2a-a013-4153-8a18-7a8aa6ec98f0-kube-api-access-9v2nl podName:0a30ff2a-a013-4153-8a18-7a8aa6ec98f0 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:34.059447326 +0000 UTC m=+4.051609583 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9v2nl" (UniqueName: "kubernetes.io/projected/0a30ff2a-a013-4153-8a18-7a8aa6ec98f0-kube-api-access-9v2nl") pod "network-check-target-mrmpn" (UID: "0a30ff2a-a013-4153-8a18-7a8aa6ec98f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 20:00:33.432181 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:33.432085    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 19:55:32 +0000 UTC" deadline="2028-05-06 12:12:16.785764274 +0000 UTC"
Sep 11 20:00:33.432181 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:33.432120    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14464h11m43.353647779s"
Sep 11 20:00:33.606912 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:33.606890    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90758be_c0bd_42cd_b5b8_65734b2b810e.slice/crio-2aed7a44be8f9cb298a6d8f374ffd2c2da193ac222b8eeb31ad8e508a294b832 WatchSource:0}: Error finding container 2aed7a44be8f9cb298a6d8f374ffd2c2da193ac222b8eeb31ad8e508a294b832: Status 404 returned error can't find the container with id 2aed7a44be8f9cb298a6d8f374ffd2c2da193ac222b8eeb31ad8e508a294b832
Sep 11 20:00:33.612135 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:33.611818    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod562cad6a_184c_4433_be65_0cd67d1a29d2.slice/crio-e1237970783865c497f2c2f1538c5d25d1ee9066d11d86a08165cdae17e3ef97 WatchSource:0}: Error finding container e1237970783865c497f2c2f1538c5d25d1ee9066d11d86a08165cdae17e3ef97: Status 404 returned error can't find the container with id e1237970783865c497f2c2f1538c5d25d1ee9066d11d86a08165cdae17e3ef97
Sep 11 20:00:33.615708 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:33.615671    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ceeee54_3eb9_4248_87ff_63b7da3896ce.slice/crio-f6517ea6212bd972b335fcad62c68105809d95ba1737d6442c94c3ba6ad54aad WatchSource:0}: Error finding container f6517ea6212bd972b335fcad62c68105809d95ba1737d6442c94c3ba6ad54aad: Status 404 returned error can't find the container with id f6517ea6212bd972b335fcad62c68105809d95ba1737d6442c94c3ba6ad54aad
Sep 11 20:00:33.616980 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:33.616944    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83199f92_0393_4012_96cb_bb5d3172786f.slice/crio-25564d93aa85c4cecfd8962c2ba242ac18d2065d8e866bdaf6bd195bd3f02428 WatchSource:0}: Error finding container 25564d93aa85c4cecfd8962c2ba242ac18d2065d8e866bdaf6bd195bd3f02428: Status 404 returned error can't find the container with id 25564d93aa85c4cecfd8962c2ba242ac18d2065d8e866bdaf6bd195bd3f02428
Sep 11 20:00:33.617967 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:33.617944    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf665fa9b_d485_4c67_8979_b4cf0c365dbb.slice/crio-2042da2003e90b44424da93733bb86ad771798a5c8d3d07e751ca0449c5a9951 WatchSource:0}: Error finding container 2042da2003e90b44424da93733bb86ad771798a5c8d3d07e751ca0449c5a9951: Status 404 returned error can't find the container with id 2042da2003e90b44424da93733bb86ad771798a5c8d3d07e751ca0449c5a9951
Sep 11 20:00:33.619740 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:33.619553    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ddb4097_dd15_4361_8302_fded613f0568.slice/crio-a9b6327e630f05bef4df03a1806de625107eca1158368af7194bc824e63750cd WatchSource:0}: Error finding container a9b6327e630f05bef4df03a1806de625107eca1158368af7194bc824e63750cd: Status 404 returned error can't find the container with id a9b6327e630f05bef4df03a1806de625107eca1158368af7194bc824e63750cd
Sep 11 20:00:33.620863 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:33.620651    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c0dbfa7_f635_4fa5_9c1a_5cd4dcd211e6.slice/crio-891977abb00da8bf84091db242a3ec0d09836aab69380e1f37773636cfcf44b0 WatchSource:0}: Error finding container 891977abb00da8bf84091db242a3ec0d09836aab69380e1f37773636cfcf44b0: Status 404 returned error can't find the container with id 891977abb00da8bf84091db242a3ec0d09836aab69380e1f37773636cfcf44b0
Sep 11 20:00:33.621598 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:33.621490    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59344c06_c093_43c1_8ef5_9e3df17dd1cd.slice/crio-4b2b8f62c7d2fc57dc4a87db3afb4a65d46cd3c864853fba70ee18860dab5192 WatchSource:0}: Error finding container 4b2b8f62c7d2fc57dc4a87db3afb4a65d46cd3c864853fba70ee18860dab5192: Status 404 returned error can't find the container with id 4b2b8f62c7d2fc57dc4a87db3afb4a65d46cd3c864853fba70ee18860dab5192
Sep 11 20:00:33.623158 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:00:33.622554    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3797d3_8b51_4fad_9bc8_54ac5e275a02.slice/crio-836d592b5af0f92446e8b717b6cc57c90b0ecebc1c1abd876374502973504beb WatchSource:0}: Error finding container 836d592b5af0f92446e8b717b6cc57c90b0ecebc1c1abd876374502973504beb: Status 404 returned error can't find the container with id 836d592b5af0f92446e8b717b6cc57c90b0ecebc1c1abd876374502973504beb
Sep 11 20:00:33.966798 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:33.966762    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda12783-5f3e-4a3e-9fde-4681797a7b60-metrics-certs\") pod \"network-metrics-daemon-fp6qw\" (UID: \"dda12783-5f3e-4a3e-9fde-4681797a7b60\") " pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:33.966962 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:33.966818    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2936adde-ce1b-4859-858d-db3584a92609-original-pull-secret\") pod \"global-pull-secret-syncer-95wcq\" (UID: \"2936adde-ce1b-4859-858d-db3584a92609\") " pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:33.966962 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:33.966912    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 20:00:33.966962 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:33.966916    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 20:00:33.967055 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:33.966969    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2936adde-ce1b-4859-858d-db3584a92609-original-pull-secret podName:2936adde-ce1b-4859-858d-db3584a92609 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:35.966953047 +0000 UTC m=+5.959115296 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2936adde-ce1b-4859-858d-db3584a92609-original-pull-secret") pod "global-pull-secret-syncer-95wcq" (UID: "2936adde-ce1b-4859-858d-db3584a92609") : object "kube-system"/"original-pull-secret" not registered
Sep 11 20:00:33.967055 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:33.966984    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda12783-5f3e-4a3e-9fde-4681797a7b60-metrics-certs podName:dda12783-5f3e-4a3e-9fde-4681797a7b60 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:35.966977224 +0000 UTC m=+5.959139472 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dda12783-5f3e-4a3e-9fde-4681797a7b60-metrics-certs") pod "network-metrics-daemon-fp6qw" (UID: "dda12783-5f3e-4a3e-9fde-4681797a7b60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 20:00:34.067668 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:34.067630    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v2nl\" (UniqueName: \"kubernetes.io/projected/0a30ff2a-a013-4153-8a18-7a8aa6ec98f0-kube-api-access-9v2nl\") pod \"network-check-target-mrmpn\" (UID: \"0a30ff2a-a013-4153-8a18-7a8aa6ec98f0\") " pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:00:34.067856 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:34.067821    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 20:00:34.067856 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:34.067840    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 20:00:34.067856 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:34.067852    2580 projected.go:196] Error preparing data for projected volume kube-api-access-9v2nl for pod openshift-network-diagnostics/network-check-target-mrmpn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 20:00:34.068015 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:34.067913    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a30ff2a-a013-4153-8a18-7a8aa6ec98f0-kube-api-access-9v2nl podName:0a30ff2a-a013-4153-8a18-7a8aa6ec98f0 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:36.067894812 +0000 UTC m=+6.060057063 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9v2nl" (UniqueName: "kubernetes.io/projected/0a30ff2a-a013-4153-8a18-7a8aa6ec98f0-kube-api-access-9v2nl") pod "network-check-target-mrmpn" (UID: "0a30ff2a-a013-4153-8a18-7a8aa6ec98f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 20:00:34.433402 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:34.433293    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 19:55:32 +0000 UTC" deadline="2028-02-20 12:12:30.092440936 +0000 UTC"
Sep 11 20:00:34.433402 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:34.433354    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12640h11m55.659091043s"
Sep 11 20:00:34.437838 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:34.437041    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:00:34.437838 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:34.437157    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrmpn" podUID="0a30ff2a-a013-4153-8a18-7a8aa6ec98f0"
Sep 11 20:00:34.437838 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:34.437546    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:34.437838 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:34.437644    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95wcq" podUID="2936adde-ce1b-4859-858d-db3584a92609"
Sep 11 20:00:34.437838 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:34.437710    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:34.437838 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:34.437789    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp6qw" podUID="dda12783-5f3e-4a3e-9fde-4681797a7b60"
Sep 11 20:00:34.454526 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:34.454450    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8nw2z" event={"ID":"562cad6a-184c-4433-be65-0cd67d1a29d2","Type":"ContainerStarted","Data":"e1237970783865c497f2c2f1538c5d25d1ee9066d11d86a08165cdae17e3ef97"}
Sep 11 20:00:34.460118 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:34.459738    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-209.ec2.internal" event={"ID":"abeccc9e8a8aea0630d3515e20a0e542","Type":"ContainerStarted","Data":"de1a082dcd30c23e8db50c35911d54e324dac2b317124fc6fccf0e18b93ce6c7"}
Sep 11 20:00:34.465626 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:34.465597    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss" event={"ID":"9c3797d3-8b51-4fad-9bc8-54ac5e275a02","Type":"ContainerStarted","Data":"836d592b5af0f92446e8b717b6cc57c90b0ecebc1c1abd876374502973504beb"}
Sep 11 20:00:34.469617 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:34.469592    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-knjv8" event={"ID":"59344c06-c093-43c1-8ef5-9e3df17dd1cd","Type":"ContainerStarted","Data":"4b2b8f62c7d2fc57dc4a87db3afb4a65d46cd3c864853fba70ee18860dab5192"}
Sep 11 20:00:34.476523 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:34.476445    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-88pnr" event={"ID":"f665fa9b-d485-4c67-8979-b4cf0c365dbb","Type":"ContainerStarted","Data":"2042da2003e90b44424da93733bb86ad771798a5c8d3d07e751ca0449c5a9951"}
Sep 11 20:00:34.485630 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:34.485566    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9n8ls" event={"ID":"4ceeee54-3eb9-4248-87ff-63b7da3896ce","Type":"ContainerStarted","Data":"f6517ea6212bd972b335fcad62c68105809d95ba1737d6442c94c3ba6ad54aad"}
Sep 11 20:00:34.495331 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:34.495261    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bpst7" event={"ID":"a90758be-c0bd-42cd-b5b8-65734b2b810e","Type":"ContainerStarted","Data":"2aed7a44be8f9cb298a6d8f374ffd2c2da193ac222b8eeb31ad8e508a294b832"}
Sep 11 20:00:34.500253 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:34.500225    2580 generic.go:356] "Generic (PLEG): container finished" podID="d75655b6d7e7980e905d4bbfadd53b9f" containerID="f5c5fa8f40b8df8241438dfc28b5dbb4d73746b06c954e4880bbea767cc0dea6" exitCode=0
Sep 11 20:00:34.500768 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:34.500515    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-209.ec2.internal" event={"ID":"d75655b6d7e7980e905d4bbfadd53b9f","Type":"ContainerDied","Data":"f5c5fa8f40b8df8241438dfc28b5dbb4d73746b06c954e4880bbea767cc0dea6"}
Sep 11 20:00:34.508803 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:34.508776    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf" event={"ID":"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6","Type":"ContainerStarted","Data":"891977abb00da8bf84091db242a3ec0d09836aab69380e1f37773636cfcf44b0"}
Sep 11 20:00:34.519511 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:34.519344    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vg4js" event={"ID":"0ddb4097-dd15-4361-8302-fded613f0568","Type":"ContainerStarted","Data":"a9b6327e630f05bef4df03a1806de625107eca1158368af7194bc824e63750cd"}
Sep 11 20:00:34.520438 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:34.520390    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-209.ec2.internal" podStartSLOduration=3.520371778 podStartE2EDuration="3.520371778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:34.475580696 +0000 UTC m=+4.467742968" watchObservedRunningTime="2026-09-11 20:00:34.520371778 +0000 UTC m=+4.512534049"
Sep 11 20:00:34.524439 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:34.524409    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j9pns" event={"ID":"83199f92-0393-4012-96cb-bb5d3172786f","Type":"ContainerStarted","Data":"25564d93aa85c4cecfd8962c2ba242ac18d2065d8e866bdaf6bd195bd3f02428"}
Sep 11 20:00:35.548915 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:35.542812    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-209.ec2.internal" event={"ID":"d75655b6d7e7980e905d4bbfadd53b9f","Type":"ContainerStarted","Data":"7a49e9ea65efc904d585286db3f2acb823e3d175640720b2a2983bbdbb1120b8"}
Sep 11 20:00:35.986788 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:35.985751    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda12783-5f3e-4a3e-9fde-4681797a7b60-metrics-certs\") pod \"network-metrics-daemon-fp6qw\" (UID: \"dda12783-5f3e-4a3e-9fde-4681797a7b60\") " pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:35.986788 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:35.985816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2936adde-ce1b-4859-858d-db3584a92609-original-pull-secret\") pod \"global-pull-secret-syncer-95wcq\" (UID: \"2936adde-ce1b-4859-858d-db3584a92609\") " pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:35.986788 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:35.985958    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 20:00:35.986788 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:35.986022    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2936adde-ce1b-4859-858d-db3584a92609-original-pull-secret podName:2936adde-ce1b-4859-858d-db3584a92609 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:39.986004596 +0000 UTC m=+9.978166850 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2936adde-ce1b-4859-858d-db3584a92609-original-pull-secret") pod "global-pull-secret-syncer-95wcq" (UID: "2936adde-ce1b-4859-858d-db3584a92609") : object "kube-system"/"original-pull-secret" not registered
Sep 11 20:00:35.986788 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:35.986648    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 20:00:35.986788 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:35.986738    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda12783-5f3e-4a3e-9fde-4681797a7b60-metrics-certs podName:dda12783-5f3e-4a3e-9fde-4681797a7b60 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:39.98672195 +0000 UTC m=+9.978884200 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dda12783-5f3e-4a3e-9fde-4681797a7b60-metrics-certs") pod "network-metrics-daemon-fp6qw" (UID: "dda12783-5f3e-4a3e-9fde-4681797a7b60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 20:00:36.087070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:36.086343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v2nl\" (UniqueName: \"kubernetes.io/projected/0a30ff2a-a013-4153-8a18-7a8aa6ec98f0-kube-api-access-9v2nl\") pod \"network-check-target-mrmpn\" (UID: \"0a30ff2a-a013-4153-8a18-7a8aa6ec98f0\") " pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:00:36.087070 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:36.086540    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 20:00:36.087070 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:36.086620    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 20:00:36.087070 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:36.086633    2580 projected.go:196] Error preparing data for projected volume kube-api-access-9v2nl for pod openshift-network-diagnostics/network-check-target-mrmpn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 20:00:36.087070 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:36.086694    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a30ff2a-a013-4153-8a18-7a8aa6ec98f0-kube-api-access-9v2nl podName:0a30ff2a-a013-4153-8a18-7a8aa6ec98f0 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:40.086673347 +0000 UTC m=+10.078835597 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9v2nl" (UniqueName: "kubernetes.io/projected/0a30ff2a-a013-4153-8a18-7a8aa6ec98f0-kube-api-access-9v2nl") pod "network-check-target-mrmpn" (UID: "0a30ff2a-a013-4153-8a18-7a8aa6ec98f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 20:00:36.437975 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:36.437124    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:00:36.437975 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:36.437249    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrmpn" podUID="0a30ff2a-a013-4153-8a18-7a8aa6ec98f0"
Sep 11 20:00:36.437975 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:36.437647    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:36.437975 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:36.437787    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95wcq" podUID="2936adde-ce1b-4859-858d-db3584a92609"
Sep 11 20:00:36.437975 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:36.437840    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:36.437975 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:36.437925    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp6qw" podUID="dda12783-5f3e-4a3e-9fde-4681797a7b60"
Sep 11 20:00:38.437749 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:38.436827    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:38.437749 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:38.436971    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp6qw" podUID="dda12783-5f3e-4a3e-9fde-4681797a7b60"
Sep 11 20:00:38.437749 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:38.437365    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:00:38.437749 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:38.437519    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrmpn" podUID="0a30ff2a-a013-4153-8a18-7a8aa6ec98f0"
Sep 11 20:00:38.437749 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:38.437579    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:38.437749 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:38.437650    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95wcq" podUID="2936adde-ce1b-4859-858d-db3584a92609"
Sep 11 20:00:40.019673 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:40.019635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda12783-5f3e-4a3e-9fde-4681797a7b60-metrics-certs\") pod \"network-metrics-daemon-fp6qw\" (UID: \"dda12783-5f3e-4a3e-9fde-4681797a7b60\") " pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:40.020128 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:40.019723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2936adde-ce1b-4859-858d-db3584a92609-original-pull-secret\") pod \"global-pull-secret-syncer-95wcq\" (UID: \"2936adde-ce1b-4859-858d-db3584a92609\") " pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:40.020128 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:40.019804    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 20:00:40.020128 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:40.019851    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 20:00:40.020128 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:40.019882    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda12783-5f3e-4a3e-9fde-4681797a7b60-metrics-certs podName:dda12783-5f3e-4a3e-9fde-4681797a7b60 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:48.019863373 +0000 UTC m=+18.012025646 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dda12783-5f3e-4a3e-9fde-4681797a7b60-metrics-certs") pod "network-metrics-daemon-fp6qw" (UID: "dda12783-5f3e-4a3e-9fde-4681797a7b60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 20:00:40.020128 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:40.019903    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2936adde-ce1b-4859-858d-db3584a92609-original-pull-secret podName:2936adde-ce1b-4859-858d-db3584a92609 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:48.019892377 +0000 UTC m=+18.012054630 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2936adde-ce1b-4859-858d-db3584a92609-original-pull-secret") pod "global-pull-secret-syncer-95wcq" (UID: "2936adde-ce1b-4859-858d-db3584a92609") : object "kube-system"/"original-pull-secret" not registered
Sep 11 20:00:40.120555 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:40.120524    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v2nl\" (UniqueName: \"kubernetes.io/projected/0a30ff2a-a013-4153-8a18-7a8aa6ec98f0-kube-api-access-9v2nl\") pod \"network-check-target-mrmpn\" (UID: \"0a30ff2a-a013-4153-8a18-7a8aa6ec98f0\") " pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:00:40.120734 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:40.120710    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 20:00:40.120789 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:40.120735    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 20:00:40.120789 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:40.120749    2580 projected.go:196] Error preparing data for projected volume kube-api-access-9v2nl for pod openshift-network-diagnostics/network-check-target-mrmpn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 20:00:40.120857 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:40.120812    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a30ff2a-a013-4153-8a18-7a8aa6ec98f0-kube-api-access-9v2nl podName:0a30ff2a-a013-4153-8a18-7a8aa6ec98f0 nodeName:}" failed. No retries permitted until 2026-09-11 20:00:48.120792951 +0000 UTC m=+18.112955206 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9v2nl" (UniqueName: "kubernetes.io/projected/0a30ff2a-a013-4153-8a18-7a8aa6ec98f0-kube-api-access-9v2nl") pod "network-check-target-mrmpn" (UID: "0a30ff2a-a013-4153-8a18-7a8aa6ec98f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 20:00:40.436696 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:40.436117    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:40.436696 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:40.436241    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:40.436696 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:40.436325    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95wcq" podUID="2936adde-ce1b-4859-858d-db3584a92609"
Sep 11 20:00:40.436696 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:40.436541    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:00:40.436696 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:40.436618    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrmpn" podUID="0a30ff2a-a013-4153-8a18-7a8aa6ec98f0"
Sep 11 20:00:40.436696 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:40.436658    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp6qw" podUID="dda12783-5f3e-4a3e-9fde-4681797a7b60"
Sep 11 20:00:42.434695 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:42.434662    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:00:42.435164 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:42.434788    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrmpn" podUID="0a30ff2a-a013-4153-8a18-7a8aa6ec98f0"
Sep 11 20:00:42.435164 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:42.434973    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:42.435164 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:42.435061    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95wcq" podUID="2936adde-ce1b-4859-858d-db3584a92609"
Sep 11 20:00:42.435164 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:42.435154    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:42.435376 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:42.435268    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp6qw" podUID="dda12783-5f3e-4a3e-9fde-4681797a7b60"
Sep 11 20:00:44.434438 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:44.434402    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:00:44.434876 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:44.434414    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:44.434876 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:44.434551    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrmpn" podUID="0a30ff2a-a013-4153-8a18-7a8aa6ec98f0"
Sep 11 20:00:44.434876 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:44.434586    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95wcq" podUID="2936adde-ce1b-4859-858d-db3584a92609"
Sep 11 20:00:44.434876 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:44.434607    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:44.434876 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:44.434736    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp6qw" podUID="dda12783-5f3e-4a3e-9fde-4681797a7b60"
Sep 11 20:00:46.435033 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:46.434837    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:46.435481 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:46.434837    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:00:46.435481 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:46.435130    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95wcq" podUID="2936adde-ce1b-4859-858d-db3584a92609"
Sep 11 20:00:46.435481 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:46.434865    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:46.435481 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:46.435210    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrmpn" podUID="0a30ff2a-a013-4153-8a18-7a8aa6ec98f0"
Sep 11 20:00:46.435481 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:46.435275    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp6qw" podUID="dda12783-5f3e-4a3e-9fde-4681797a7b60"
Sep 11 20:00:48.080946 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:48.080903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda12783-5f3e-4a3e-9fde-4681797a7b60-metrics-certs\") pod \"network-metrics-daemon-fp6qw\" (UID: \"dda12783-5f3e-4a3e-9fde-4681797a7b60\") " pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:48.081404 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:48.080979    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2936adde-ce1b-4859-858d-db3584a92609-original-pull-secret\") pod \"global-pull-secret-syncer-95wcq\" (UID: \"2936adde-ce1b-4859-858d-db3584a92609\") " pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:48.081404 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:48.081068    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 20:00:48.081404 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:48.081125    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2936adde-ce1b-4859-858d-db3584a92609-original-pull-secret podName:2936adde-ce1b-4859-858d-db3584a92609 nodeName:}" failed. No retries permitted until 2026-09-11 20:01:04.081110776 +0000 UTC m=+34.073273024 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2936adde-ce1b-4859-858d-db3584a92609-original-pull-secret") pod "global-pull-secret-syncer-95wcq" (UID: "2936adde-ce1b-4859-858d-db3584a92609") : object "kube-system"/"original-pull-secret" not registered
Sep 11 20:00:48.081404 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:48.081068    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 20:00:48.081404 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:48.081239    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda12783-5f3e-4a3e-9fde-4681797a7b60-metrics-certs podName:dda12783-5f3e-4a3e-9fde-4681797a7b60 nodeName:}" failed. No retries permitted until 2026-09-11 20:01:04.08121616 +0000 UTC m=+34.073378424 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dda12783-5f3e-4a3e-9fde-4681797a7b60-metrics-certs") pod "network-metrics-daemon-fp6qw" (UID: "dda12783-5f3e-4a3e-9fde-4681797a7b60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 20:00:48.182078 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:48.182039    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v2nl\" (UniqueName: \"kubernetes.io/projected/0a30ff2a-a013-4153-8a18-7a8aa6ec98f0-kube-api-access-9v2nl\") pod \"network-check-target-mrmpn\" (UID: \"0a30ff2a-a013-4153-8a18-7a8aa6ec98f0\") " pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:00:48.182244 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:48.182184    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 20:00:48.182244 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:48.182205    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 20:00:48.182244 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:48.182215    2580 projected.go:196] Error preparing data for projected volume kube-api-access-9v2nl for pod openshift-network-diagnostics/network-check-target-mrmpn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 20:00:48.182386 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:48.182268    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a30ff2a-a013-4153-8a18-7a8aa6ec98f0-kube-api-access-9v2nl podName:0a30ff2a-a013-4153-8a18-7a8aa6ec98f0 nodeName:}" failed. No retries permitted until 2026-09-11 20:01:04.182251776 +0000 UTC m=+34.174414039 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9v2nl" (UniqueName: "kubernetes.io/projected/0a30ff2a-a013-4153-8a18-7a8aa6ec98f0-kube-api-access-9v2nl") pod "network-check-target-mrmpn" (UID: "0a30ff2a-a013-4153-8a18-7a8aa6ec98f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 20:00:48.435311 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:48.435283    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:48.435508 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:48.435331    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:48.435508 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:48.435421    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp6qw" podUID="dda12783-5f3e-4a3e-9fde-4681797a7b60"
Sep 11 20:00:48.435628 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:48.435480    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95wcq" podUID="2936adde-ce1b-4859-858d-db3584a92609"
Sep 11 20:00:48.435628 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:48.435541    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:00:48.435709 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:48.435630    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrmpn" podUID="0a30ff2a-a013-4153-8a18-7a8aa6ec98f0"
Sep 11 20:00:50.435154 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:50.435120    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:00:50.435616 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:50.435197    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:50.435616 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:50.435208    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrmpn" podUID="0a30ff2a-a013-4153-8a18-7a8aa6ec98f0"
Sep 11 20:00:50.435616 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:50.435300    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95wcq" podUID="2936adde-ce1b-4859-858d-db3584a92609"
Sep 11 20:00:50.435616 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:50.435366    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:50.435616 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:50.435505    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp6qw" podUID="dda12783-5f3e-4a3e-9fde-4681797a7b60"
Sep 11 20:00:51.577905 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:51.577670    2580 generic.go:356] "Generic (PLEG): container finished" podID="4ceeee54-3eb9-4248-87ff-63b7da3896ce" containerID="86b1c0b5b265acb22ef2139781045c4fd921ae9a8501691bb947b1ee3d917857" exitCode=0
Sep 11 20:00:51.578601 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:51.577753    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9n8ls" event={"ID":"4ceeee54-3eb9-4248-87ff-63b7da3896ce","Type":"ContainerDied","Data":"86b1c0b5b265acb22ef2139781045c4fd921ae9a8501691bb947b1ee3d917857"}
Sep 11 20:00:51.579540 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:51.579522    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bpst7" event={"ID":"a90758be-c0bd-42cd-b5b8-65734b2b810e","Type":"ContainerStarted","Data":"660c2a5b700d31c24b4a81ff1321094068103bffbf5e7522e2da6343affd92af"}
Sep 11 20:00:51.582005 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:51.581986    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf" event={"ID":"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6","Type":"ContainerStarted","Data":"ba544b42aa6e2bb964517722230e94ee104ffdb1ed598655ddc8261179feba6a"}
Sep 11 20:00:51.582093 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:51.582044    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf" event={"ID":"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6","Type":"ContainerStarted","Data":"d1ea7a3969b6f9d81ba9f335018c5695a0fd66b7b6d9bbfd994a3309df84bc5e"}
Sep 11 20:00:51.582093 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:51.582088    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf" event={"ID":"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6","Type":"ContainerStarted","Data":"afd5de3c008efe687fcf511af04b0622fd577767fb020c8fd8e9c4c2608dd364"}
Sep 11 20:00:51.582167 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:51.582154    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf" event={"ID":"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6","Type":"ContainerStarted","Data":"9f0d4fb433c57ec773eb2840a31835791525d28669d82d2c76471b83f0dca43c"}
Sep 11 20:00:51.583689 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:51.583670    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vg4js" event={"ID":"0ddb4097-dd15-4361-8302-fded613f0568","Type":"ContainerStarted","Data":"ccbf03b351b9fd6234b43892258f9fbc55fe037372a66b5f49303060304f0cae"}
Sep 11 20:00:51.584917 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:51.584897    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j9pns" event={"ID":"83199f92-0393-4012-96cb-bb5d3172786f","Type":"ContainerStarted","Data":"2147ed8ece8ec2b2cea710a4e91ebc073faa2cee54c1db60acf13f0cb4dbd9b7"}
Sep 11 20:00:51.586170 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:51.586152    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8nw2z" event={"ID":"562cad6a-184c-4433-be65-0cd67d1a29d2","Type":"ContainerStarted","Data":"7df2b7b8c6e712aa991ad7d4e78bd941dfd26a16f949d33f28996877a77f86b8"}
Sep 11 20:00:51.587452 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:51.587432    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss" event={"ID":"9c3797d3-8b51-4fad-9bc8-54ac5e275a02","Type":"ContainerStarted","Data":"26ff084f572fc7717d4eeaf8c2ded54dfda3c4056e658867602f1bea056781a3"}
Sep 11 20:00:51.588722 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:51.588701    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-knjv8" event={"ID":"59344c06-c093-43c1-8ef5-9e3df17dd1cd","Type":"ContainerStarted","Data":"96cd03bfe7c7cec437258cdd24b8d518554788a77552b8aff7ff85242ee06405"}
Sep 11 20:00:51.599543 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:51.599483    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-209.ec2.internal" podStartSLOduration=20.599471725 podStartE2EDuration="20.599471725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:35.560557978 +0000 UTC m=+5.552720249" watchObservedRunningTime="2026-09-11 20:00:51.599471725 +0000 UTC m=+21.591633995"
Sep 11 20:00:51.612940 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:51.612905    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-knjv8" podStartSLOduration=4.117054814 podStartE2EDuration="21.612894445s" totalImagesPullingTime="17.495839631s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:51.612575933 +0000 UTC m=+21.604738203" watchObservedRunningTime="2026-09-11 20:00:51.612894445 +0000 UTC m=+21.605056712"
Sep 11 20:00:51.630172 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:51.630133    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-j9pns" podStartSLOduration=4.129420276 podStartE2EDuration="21.630117487s" totalImagesPullingTime="17.500697211s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:51.6297452 +0000 UTC m=+21.621907470" watchObservedRunningTime="2026-09-11 20:00:51.630117487 +0000 UTC m=+21.622279757"
Sep 11 20:00:51.645053 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:51.645022    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vg4js" podStartSLOduration=11.424251457 podStartE2EDuration="21.645010272s" totalImagesPullingTime="10.220758815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:51.644638377 +0000 UTC m=+21.636800648" watchObservedRunningTime="2026-09-11 20:00:51.645010272 +0000 UTC m=+21.637172541"
Sep 11 20:00:51.663989 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:51.663966    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vg4js"
Sep 11 20:00:51.664410 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:51.664396    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vg4js"
Sep 11 20:00:51.671816 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:51.671783    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bpst7" podStartSLOduration=4.162810206 podStartE2EDuration="21.671771472s" totalImagesPullingTime="17.508961266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:51.671235457 +0000 UTC m=+21.663397718" watchObservedRunningTime="2026-09-11 20:00:51.671771472 +0000 UTC m=+21.663933741"
Sep 11 20:00:51.688140 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:51.688104    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8nw2z" podStartSLOduration=4.12448393 podStartE2EDuration="21.688093045s" totalImagesPullingTime="17.563609115s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:51.687859186 +0000 UTC m=+21.680021452" watchObservedRunningTime="2026-09-11 20:00:51.688093045 +0000 UTC m=+21.680255315"
Sep 11 20:00:52.435062 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:52.435031    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:52.435232 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:52.435063    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:00:52.435232 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:52.435032    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:52.435232 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:52.435168    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp6qw" podUID="dda12783-5f3e-4a3e-9fde-4681797a7b60"
Sep 11 20:00:52.435386 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:52.435276    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrmpn" podUID="0a30ff2a-a013-4153-8a18-7a8aa6ec98f0"
Sep 11 20:00:52.435436 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:52.435378    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95wcq" podUID="2936adde-ce1b-4859-858d-db3584a92609"
Sep 11 20:00:52.599684 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:52.599646    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf" event={"ID":"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6","Type":"ContainerStarted","Data":"b1389dc7ad3350809672004a07a64aa987c7909814e0696a8ef5db013e4461f0"}
Sep 11 20:00:52.600201 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:52.599764    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf" event={"ID":"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6","Type":"ContainerStarted","Data":"f0d0628c426257749ce95850aaae4d203ac1e14524ab0d629347a488f7d17e29"}
Sep 11 20:00:52.601936 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:52.601854    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-88pnr" event={"ID":"f665fa9b-d485-4c67-8979-b4cf0c365dbb","Type":"ContainerStarted","Data":"6f205c1b53b9f1e7fccb85373bfe7cbf89638c22396c2e0c5f7aed18ce2ad6a8"}
Sep 11 20:00:52.602851 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:52.602769    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vg4js"
Sep 11 20:00:52.603375 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:52.603356    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vg4js"
Sep 11 20:00:52.622534 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:52.622480    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-88pnr" podStartSLOduration=5.122461232 podStartE2EDuration="22.622466003s" totalImagesPullingTime="17.500004771s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:52.621827251 +0000 UTC m=+22.613989522" watchObservedRunningTime="2026-09-11 20:00:52.622466003 +0000 UTC m=+22.614628264"
Sep 11 20:00:53.152663 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:53.152641    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 20:00:53.390577 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:53.390399    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T20:00:53.152660244Z","UUID":"e877d966-2391-4465-9b9b-18bac849f3d2","Handler":null,"Name":"","Endpoint":""}
Sep 11 20:00:53.392093 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:53.392065    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 20:00:53.392231 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:53.392100    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 20:00:53.606300 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:53.606266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss" event={"ID":"9c3797d3-8b51-4fad-9bc8-54ac5e275a02","Type":"ContainerStarted","Data":"57f5d238af7fc0bd53bb4acecfb9ad3d52821e01439a773a17aa1f713b762142"}
Sep 11 20:00:54.435107 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:54.435078    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:00:54.435398 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:54.435193    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrmpn" podUID="0a30ff2a-a013-4153-8a18-7a8aa6ec98f0"
Sep 11 20:00:54.435398 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:54.435087    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:54.435398 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:54.435230    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:54.435398 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:54.435319    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95wcq" podUID="2936adde-ce1b-4859-858d-db3584a92609"
Sep 11 20:00:54.435595 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:54.435411    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp6qw" podUID="dda12783-5f3e-4a3e-9fde-4681797a7b60"
Sep 11 20:00:54.612051 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:54.612019    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf" event={"ID":"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6","Type":"ContainerStarted","Data":"48b64c979c333a53e15d1ef8d64fbe5f955e6be8f02376d706bb0a79abf8c042"}
Sep 11 20:00:55.617213 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:55.617174    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss" event={"ID":"9c3797d3-8b51-4fad-9bc8-54ac5e275a02","Type":"ContainerStarted","Data":"4293c38c1fc1707bbfe7c062a4c7eb10e5b969b284dccbf9c750baaeb057994f"}
Sep 11 20:00:55.636008 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:55.635960    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9gss" podStartSLOduration=5.076126954 podStartE2EDuration="25.635944297s" totalImagesPullingTime="20.559817343s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:00:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:55.635556231 +0000 UTC m=+25.627718502" watchObservedRunningTime="2026-09-11 20:00:55.635944297 +0000 UTC m=+25.628106567"
Sep 11 20:00:56.434676 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:56.434454    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:00:56.434818 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:56.434473    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:56.434818 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:56.434759    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrmpn" podUID="0a30ff2a-a013-4153-8a18-7a8aa6ec98f0"
Sep 11 20:00:56.434884 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:56.434529    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:56.434916 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:56.434820    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95wcq" podUID="2936adde-ce1b-4859-858d-db3584a92609"
Sep 11 20:00:56.434954 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:56.434924    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp6qw" podUID="dda12783-5f3e-4a3e-9fde-4681797a7b60"
Sep 11 20:00:56.620797 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:56.620715    2580 generic.go:356] "Generic (PLEG): container finished" podID="4ceeee54-3eb9-4248-87ff-63b7da3896ce" containerID="b36332e6e7c898f0b1a18b9851a9f2bce39a5d393ff34476becdd9c8302cbe08" exitCode=0
Sep 11 20:00:56.621291 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:56.620802    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9n8ls" event={"ID":"4ceeee54-3eb9-4248-87ff-63b7da3896ce","Type":"ContainerDied","Data":"b36332e6e7c898f0b1a18b9851a9f2bce39a5d393ff34476becdd9c8302cbe08"}
Sep 11 20:00:56.624844 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:56.624807    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf" event={"ID":"5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6","Type":"ContainerStarted","Data":"7b32fbce2ead8a689fe32e65c94f91007f089254b91a0568529bb891ef916749"}
Sep 11 20:00:56.670792 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:56.670734    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf" podStartSLOduration=9.14464679 podStartE2EDuration="26.670719318s" totalImagesPullingTime="17.526072528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:56.670088128 +0000 UTC m=+26.662250408" watchObservedRunningTime="2026-09-11 20:00:56.670719318 +0000 UTC m=+26.662881585"
Sep 11 20:00:57.629509 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:57.629460    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9n8ls" event={"ID":"4ceeee54-3eb9-4248-87ff-63b7da3896ce","Type":"ContainerStarted","Data":"b7c9ac856431819712007622ff0f381c78300088fc1dcc20f5a1e137515dde8f"}
Sep 11 20:00:57.630152 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:57.630133    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:57.630260 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:57.630202    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:57.630260 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:57.630246    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:57.647177 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:57.647149    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:57.647318 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:57.647294    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:00:58.058402 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:58.058328    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fp6qw"]
Sep 11 20:00:58.058551 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:58.058451    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:58.058600 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:58.058565    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp6qw" podUID="dda12783-5f3e-4a3e-9fde-4681797a7b60"
Sep 11 20:00:58.058928 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:58.058906    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-95wcq"]
Sep 11 20:00:58.059002 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:58.058992    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:58.059073 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:58.059059    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95wcq" podUID="2936adde-ce1b-4859-858d-db3584a92609"
Sep 11 20:00:58.066015 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:58.065992    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mrmpn"]
Sep 11 20:00:58.066111 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:58.066072    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:00:58.066148 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:58.066134    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrmpn" podUID="0a30ff2a-a013-4153-8a18-7a8aa6ec98f0"
Sep 11 20:00:58.633776 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:58.633746    2580 generic.go:356] "Generic (PLEG): container finished" podID="4ceeee54-3eb9-4248-87ff-63b7da3896ce" containerID="b7c9ac856431819712007622ff0f381c78300088fc1dcc20f5a1e137515dde8f" exitCode=0
Sep 11 20:00:58.634395 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:58.633830    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9n8ls" event={"ID":"4ceeee54-3eb9-4248-87ff-63b7da3896ce","Type":"ContainerDied","Data":"b7c9ac856431819712007622ff0f381c78300088fc1dcc20f5a1e137515dde8f"}
Sep 11 20:00:59.435160 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:59.435049    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:00:59.435160 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:59.435077    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:00:59.435160 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:59.435049    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:00:59.435412 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:59.435186    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp6qw" podUID="dda12783-5f3e-4a3e-9fde-4681797a7b60"
Sep 11 20:00:59.435412 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:59.435290    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95wcq" podUID="2936adde-ce1b-4859-858d-db3584a92609"
Sep 11 20:00:59.435412 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:00:59.435372    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrmpn" podUID="0a30ff2a-a013-4153-8a18-7a8aa6ec98f0"
Sep 11 20:00:59.638627 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:59.638600    2580 generic.go:356] "Generic (PLEG): container finished" podID="4ceeee54-3eb9-4248-87ff-63b7da3896ce" containerID="b9ac37abdd32e03561ba436512fbecf341f4c3d63cae9e39dff2b93d221fc909" exitCode=0
Sep 11 20:00:59.639093 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:00:59.638685    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9n8ls" event={"ID":"4ceeee54-3eb9-4248-87ff-63b7da3896ce","Type":"ContainerDied","Data":"b9ac37abdd32e03561ba436512fbecf341f4c3d63cae9e39dff2b93d221fc909"}
Sep 11 20:01:00.997744 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:00.997720    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j9pns_83199f92-0393-4012-96cb-bb5d3172786f/dns-node-resolver/0.log"
Sep 11 20:01:01.434461 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:01.434430    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:01:01.434642 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:01.434465    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:01:01.434642 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:01.434578    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:01:01.434642 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:01:01.434591    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp6qw" podUID="dda12783-5f3e-4a3e-9fde-4681797a7b60"
Sep 11 20:01:01.434810 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:01:01.434674    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95wcq" podUID="2936adde-ce1b-4859-858d-db3584a92609"
Sep 11 20:01:01.434810 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:01:01.434750    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrmpn" podUID="0a30ff2a-a013-4153-8a18-7a8aa6ec98f0"
Sep 11 20:01:01.978188 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:01.978162    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-knjv8_59344c06-c093-43c1-8ef5-9e3df17dd1cd/node-ca/0.log"
Sep 11 20:01:03.434632 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:03.434593    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:01:03.435238 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:03.434640    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:01:03.435238 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:03.434640    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:01:03.435238 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:01:03.434716    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrmpn" podUID="0a30ff2a-a013-4153-8a18-7a8aa6ec98f0"
Sep 11 20:01:03.435238 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:01:03.434808    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95wcq" podUID="2936adde-ce1b-4859-858d-db3584a92609"
Sep 11 20:01:03.435238 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:01:03.434875    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fp6qw" podUID="dda12783-5f3e-4a3e-9fde-4681797a7b60"
Sep 11 20:01:04.096792 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:04.096758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda12783-5f3e-4a3e-9fde-4681797a7b60-metrics-certs\") pod \"network-metrics-daemon-fp6qw\" (UID: \"dda12783-5f3e-4a3e-9fde-4681797a7b60\") " pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:01:04.096982 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:04.096819    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2936adde-ce1b-4859-858d-db3584a92609-original-pull-secret\") pod \"global-pull-secret-syncer-95wcq\" (UID: \"2936adde-ce1b-4859-858d-db3584a92609\") " pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:01:04.096982 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:01:04.096914    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 20:01:04.096982 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:01:04.096921    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 20:01:04.096982 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:01:04.096956    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2936adde-ce1b-4859-858d-db3584a92609-original-pull-secret podName:2936adde-ce1b-4859-858d-db3584a92609 nodeName:}" failed. No retries permitted until 2026-09-11 20:01:36.096943991 +0000 UTC m=+66.089106239 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2936adde-ce1b-4859-858d-db3584a92609-original-pull-secret") pod "global-pull-secret-syncer-95wcq" (UID: "2936adde-ce1b-4859-858d-db3584a92609") : object "kube-system"/"original-pull-secret" not registered
Sep 11 20:01:04.096982 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:01:04.096978    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda12783-5f3e-4a3e-9fde-4681797a7b60-metrics-certs podName:dda12783-5f3e-4a3e-9fde-4681797a7b60 nodeName:}" failed. No retries permitted until 2026-09-11 20:01:36.096961329 +0000 UTC m=+66.089123576 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dda12783-5f3e-4a3e-9fde-4681797a7b60-metrics-certs") pod "network-metrics-daemon-fp6qw" (UID: "dda12783-5f3e-4a3e-9fde-4681797a7b60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 20:01:04.197413 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:04.197383    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v2nl\" (UniqueName: \"kubernetes.io/projected/0a30ff2a-a013-4153-8a18-7a8aa6ec98f0-kube-api-access-9v2nl\") pod \"network-check-target-mrmpn\" (UID: \"0a30ff2a-a013-4153-8a18-7a8aa6ec98f0\") " pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:01:04.197591 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:01:04.197547    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 20:01:04.197591 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:01:04.197565    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 20:01:04.197591 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:01:04.197575    2580 projected.go:196] Error preparing data for projected volume kube-api-access-9v2nl for pod openshift-network-diagnostics/network-check-target-mrmpn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 20:01:04.197732 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:01:04.197631    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0a30ff2a-a013-4153-8a18-7a8aa6ec98f0-kube-api-access-9v2nl podName:0a30ff2a-a013-4153-8a18-7a8aa6ec98f0 nodeName:}" failed. No retries permitted until 2026-09-11 20:01:36.197617378 +0000 UTC m=+66.189779626 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9v2nl" (UniqueName: "kubernetes.io/projected/0a30ff2a-a013-4153-8a18-7a8aa6ec98f0-kube-api-access-9v2nl") pod "network-check-target-mrmpn" (UID: "0a30ff2a-a013-4153-8a18-7a8aa6ec98f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 20:01:04.785763 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:04.785737    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-209.ec2.internal" event="NodeReady"
Sep 11 20:01:04.786071 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:04.785891    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 20:01:05.435330 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:05.435301    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:01:05.435488 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:05.435301    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:01:05.435488 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:05.435301    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:01:05.438107 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:05.438087    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 20:01:05.438208 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:05.438128    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 20:01:05.439709 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:05.439694    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 20:01:05.439804 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:05.439728    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 20:01:05.439804 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:05.439742    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b7zd7\""
Sep 11 20:01:05.439916 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:05.439816    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fv8gc\""
Sep 11 20:01:05.654357 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:05.654328    2580 generic.go:356] "Generic (PLEG): container finished" podID="4ceeee54-3eb9-4248-87ff-63b7da3896ce" containerID="9325dd11f074090e2d57e7541564690c8eb1bbebe6408ee06738722213122de8" exitCode=0
Sep 11 20:01:05.654526 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:05.654375    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9n8ls" event={"ID":"4ceeee54-3eb9-4248-87ff-63b7da3896ce","Type":"ContainerDied","Data":"9325dd11f074090e2d57e7541564690c8eb1bbebe6408ee06738722213122de8"}
Sep 11 20:01:06.660225 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:06.660016    2580 generic.go:356] "Generic (PLEG): container finished" podID="4ceeee54-3eb9-4248-87ff-63b7da3896ce" containerID="f04e5a4e4218db9b08b84706fd15a64fb1babc7855ebf9435dbcd4d2d5dd5f41" exitCode=0
Sep 11 20:01:06.660225 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:06.660085    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9n8ls" event={"ID":"4ceeee54-3eb9-4248-87ff-63b7da3896ce","Type":"ContainerDied","Data":"f04e5a4e4218db9b08b84706fd15a64fb1babc7855ebf9435dbcd4d2d5dd5f41"}
Sep 11 20:01:07.666525 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:07.666475    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9n8ls" event={"ID":"4ceeee54-3eb9-4248-87ff-63b7da3896ce","Type":"ContainerStarted","Data":"9f1e335031f1d72729cd94e49d10338bf754430b004e46503f90c1757f5e03d3"}
Sep 11 20:01:07.690152 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:07.690108    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9n8ls" podStartSLOduration=8.931956787 podStartE2EDuration="37.690094555s" totalImagesPullingTime="28.758137768s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:00:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:07.689802633 +0000 UTC m=+37.681964913" watchObservedRunningTime="2026-09-11 20:01:07.690094555 +0000 UTC m=+37.682256825"
Sep 11 20:01:22.976151 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:22.976117    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78b79fd667-jpnm6"]
Sep 11 20:01:22.978701 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:22.978680    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:22.982272 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:22.982246    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 20:01:22.982272 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:22.982278    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 20:01:22.982537 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:22.982519    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l4rd2\""
Sep 11 20:01:22.983736 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:22.983720    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 20:01:22.987462 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:22.987444    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 20:01:23.005830 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.005807    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78b79fd667-jpnm6"]
Sep 11 20:01:23.005968 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:01:23.005945    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-gshlb registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-gshlb registry-certificates registry-tls trusted-ca]: context canceled" pod="openshift-image-registry/image-registry-78b79fd667-jpnm6" podUID="6b755347-8d4f-42bb-be5b-fff48cb4ac46"
Sep 11 20:01:23.034050 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.034027    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78b79fd667-jpnm6"]
Sep 11 20:01:23.046324 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.046302    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-wzwg7"]
Sep 11 20:01:23.048253 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.048238    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cchrc"]
Sep 11 20:01:23.048383 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.048367    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-wzwg7"
Sep 11 20:01:23.050028 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.050011    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cchrc"
Sep 11 20:01:23.050966 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.050930    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-trmvh\""
Sep 11 20:01:23.051674 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.051654    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 20:01:23.051775 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.051681    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 20:01:23.052105 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.052086    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68887c9c9b-thsrj"]
Sep 11 20:01:23.052203 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.052123    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 20:01:23.052655 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.052639    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 20:01:23.052914 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.052902    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zqht8\""
Sep 11 20:01:23.053374 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.053358    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 20:01:23.053635 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.053611    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 20:01:23.054218 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.054203    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.064131 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.064113    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cchrc"]
Sep 11 20:01:23.064928 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.064908    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-wzwg7"]
Sep 11 20:01:23.070105 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.070083    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68887c9c9b-thsrj"]
Sep 11 20:01:23.120399 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.120375    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b755347-8d4f-42bb-be5b-fff48cb4ac46-registry-tls\") pod \"image-registry-78b79fd667-jpnm6\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") " pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.120560 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.120422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gshlb\" (UniqueName: \"kubernetes.io/projected/6b755347-8d4f-42bb-be5b-fff48cb4ac46-kube-api-access-gshlb\") pod \"image-registry-78b79fd667-jpnm6\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") " pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.120560 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.120442    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b755347-8d4f-42bb-be5b-fff48cb4ac46-image-registry-private-configuration\") pod \"image-registry-78b79fd667-jpnm6\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") " pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.120560 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.120514    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b755347-8d4f-42bb-be5b-fff48cb4ac46-trusted-ca\") pod \"image-registry-78b79fd667-jpnm6\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") " pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.120732 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.120565    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b755347-8d4f-42bb-be5b-fff48cb4ac46-installation-pull-secrets\") pod \"image-registry-78b79fd667-jpnm6\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") " pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.120732 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.120609    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b755347-8d4f-42bb-be5b-fff48cb4ac46-registry-certificates\") pod \"image-registry-78b79fd667-jpnm6\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") " pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.120732 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.120635    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b755347-8d4f-42bb-be5b-fff48cb4ac46-bound-sa-token\") pod \"image-registry-78b79fd667-jpnm6\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") " pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.120732 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.120676    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b755347-8d4f-42bb-be5b-fff48cb4ac46-ca-trust-extracted\") pod \"image-registry-78b79fd667-jpnm6\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") " pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.150650 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.150626    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-q484p"]
Sep 11 20:01:23.152349 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.152334    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rssdb"]
Sep 11 20:01:23.152488 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.152470    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q484p"
Sep 11 20:01:23.154372 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.154353    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rssdb"
Sep 11 20:01:23.155969 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.155950    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 20:01:23.156061 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.155980    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nbwp6\""
Sep 11 20:01:23.156124 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.156084    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 20:01:23.156966 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.156951    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 20:01:23.156966 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.156961    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fjh5z\""
Sep 11 20:01:23.157105 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.156985    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 20:01:23.157159 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.157121    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 20:01:23.169085 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.169061    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rssdb"]
Sep 11 20:01:23.169731 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.169711    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q484p"]
Sep 11 20:01:23.221434 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.221408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txvzc\" (UniqueName: \"kubernetes.io/projected/df290aa2-b262-4940-9eea-054aca381519-kube-api-access-txvzc\") pod \"downloads-76f99584f4-wzwg7\" (UID: \"df290aa2-b262-4940-9eea-054aca381519\") " pod="openshift-console/downloads-76f99584f4-wzwg7"
Sep 11 20:01:23.221554 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.221439    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1819bd16-803f-4ec5-a141-5911232d0799-registry-tls\") pod \"image-registry-68887c9c9b-thsrj\" (UID: \"1819bd16-803f-4ec5-a141-5911232d0799\") " pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.221554 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.221460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b755347-8d4f-42bb-be5b-fff48cb4ac46-ca-trust-extracted\") pod \"image-registry-78b79fd667-jpnm6\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") " pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.221554 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.221477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1819bd16-803f-4ec5-a141-5911232d0799-image-registry-private-configuration\") pod \"image-registry-68887c9c9b-thsrj\" (UID: \"1819bd16-803f-4ec5-a141-5911232d0799\") " pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.221554 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.221505    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1819bd16-803f-4ec5-a141-5911232d0799-installation-pull-secrets\") pod \"image-registry-68887c9c9b-thsrj\" (UID: \"1819bd16-803f-4ec5-a141-5911232d0799\") " pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.221702 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.221553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1819bd16-803f-4ec5-a141-5911232d0799-bound-sa-token\") pod \"image-registry-68887c9c9b-thsrj\" (UID: \"1819bd16-803f-4ec5-a141-5911232d0799\") " pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.221702 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.221590    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2197b3ff-694b-4b17-9916-149ec1eb9924-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cchrc\" (UID: \"2197b3ff-694b-4b17-9916-149ec1eb9924\") " pod="openshift-insights/insights-runtime-extractor-cchrc"
Sep 11 20:01:23.221702 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.221647    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1819bd16-803f-4ec5-a141-5911232d0799-registry-certificates\") pod \"image-registry-68887c9c9b-thsrj\" (UID: \"1819bd16-803f-4ec5-a141-5911232d0799\") " pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.221702 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.221667    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kf6d\" (UniqueName: \"kubernetes.io/projected/2197b3ff-694b-4b17-9916-149ec1eb9924-kube-api-access-9kf6d\") pod \"insights-runtime-extractor-cchrc\" (UID: \"2197b3ff-694b-4b17-9916-149ec1eb9924\") " pod="openshift-insights/insights-runtime-extractor-cchrc"
Sep 11 20:01:23.221702 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.221686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b755347-8d4f-42bb-be5b-fff48cb4ac46-registry-tls\") pod \"image-registry-78b79fd667-jpnm6\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") " pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.221873 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.221715    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klvb7\" (UniqueName: \"kubernetes.io/projected/1819bd16-803f-4ec5-a141-5911232d0799-kube-api-access-klvb7\") pod \"image-registry-68887c9c9b-thsrj\" (UID: \"1819bd16-803f-4ec5-a141-5911232d0799\") " pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.221873 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.221743    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gshlb\" (UniqueName: \"kubernetes.io/projected/6b755347-8d4f-42bb-be5b-fff48cb4ac46-kube-api-access-gshlb\") pod \"image-registry-78b79fd667-jpnm6\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") " pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.221873 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.221758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1819bd16-803f-4ec5-a141-5911232d0799-ca-trust-extracted\") pod \"image-registry-68887c9c9b-thsrj\" (UID: \"1819bd16-803f-4ec5-a141-5911232d0799\") " pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.221873 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.221814    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b755347-8d4f-42bb-be5b-fff48cb4ac46-ca-trust-extracted\") pod \"image-registry-78b79fd667-jpnm6\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") " pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.221873 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.221855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1819bd16-803f-4ec5-a141-5911232d0799-trusted-ca\") pod \"image-registry-68887c9c9b-thsrj\" (UID: \"1819bd16-803f-4ec5-a141-5911232d0799\") " pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.222035 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.221912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b755347-8d4f-42bb-be5b-fff48cb4ac46-image-registry-private-configuration\") pod \"image-registry-78b79fd667-jpnm6\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") " pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.222035 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.221952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b755347-8d4f-42bb-be5b-fff48cb4ac46-trusted-ca\") pod \"image-registry-78b79fd667-jpnm6\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") " pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.222035 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.221973    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b755347-8d4f-42bb-be5b-fff48cb4ac46-installation-pull-secrets\") pod \"image-registry-78b79fd667-jpnm6\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") " pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.222194 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.222167    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2197b3ff-694b-4b17-9916-149ec1eb9924-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cchrc\" (UID: \"2197b3ff-694b-4b17-9916-149ec1eb9924\") " pod="openshift-insights/insights-runtime-extractor-cchrc"
Sep 11 20:01:23.222315 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.222209    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2197b3ff-694b-4b17-9916-149ec1eb9924-crio-socket\") pod \"insights-runtime-extractor-cchrc\" (UID: \"2197b3ff-694b-4b17-9916-149ec1eb9924\") " pod="openshift-insights/insights-runtime-extractor-cchrc"
Sep 11 20:01:23.222315 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.222236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b755347-8d4f-42bb-be5b-fff48cb4ac46-registry-certificates\") pod \"image-registry-78b79fd667-jpnm6\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") " pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.222315 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.222257    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b755347-8d4f-42bb-be5b-fff48cb4ac46-bound-sa-token\") pod \"image-registry-78b79fd667-jpnm6\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") " pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.222315 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.222280    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2197b3ff-694b-4b17-9916-149ec1eb9924-data-volume\") pod \"insights-runtime-extractor-cchrc\" (UID: \"2197b3ff-694b-4b17-9916-149ec1eb9924\") " pod="openshift-insights/insights-runtime-extractor-cchrc"
Sep 11 20:01:23.223120 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.223094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b755347-8d4f-42bb-be5b-fff48cb4ac46-registry-certificates\") pod \"image-registry-78b79fd667-jpnm6\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") " pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.223213 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.223097    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b755347-8d4f-42bb-be5b-fff48cb4ac46-trusted-ca\") pod \"image-registry-78b79fd667-jpnm6\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") " pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.225946 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.225920    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b755347-8d4f-42bb-be5b-fff48cb4ac46-registry-tls\") pod \"image-registry-78b79fd667-jpnm6\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") " pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.226050 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.225990    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b755347-8d4f-42bb-be5b-fff48cb4ac46-installation-pull-secrets\") pod \"image-registry-78b79fd667-jpnm6\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") " pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.226050 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.226024    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b755347-8d4f-42bb-be5b-fff48cb4ac46-image-registry-private-configuration\") pod \"image-registry-78b79fd667-jpnm6\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") " pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.240309 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.240262    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b755347-8d4f-42bb-be5b-fff48cb4ac46-bound-sa-token\") pod \"image-registry-78b79fd667-jpnm6\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") " pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.243153 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.243124    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gshlb\" (UniqueName: \"kubernetes.io/projected/6b755347-8d4f-42bb-be5b-fff48cb4ac46-kube-api-access-gshlb\") pod \"image-registry-78b79fd667-jpnm6\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") " pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.323204 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.323182    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1819bd16-803f-4ec5-a141-5911232d0799-ca-trust-extracted\") pod \"image-registry-68887c9c9b-thsrj\" (UID: \"1819bd16-803f-4ec5-a141-5911232d0799\") " pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.323204 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.323206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1819bd16-803f-4ec5-a141-5911232d0799-trusted-ca\") pod \"image-registry-68887c9c9b-thsrj\" (UID: \"1819bd16-803f-4ec5-a141-5911232d0799\") " pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.323353 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.323235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2197b3ff-694b-4b17-9916-149ec1eb9924-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cchrc\" (UID: \"2197b3ff-694b-4b17-9916-149ec1eb9924\") " pod="openshift-insights/insights-runtime-extractor-cchrc"
Sep 11 20:01:23.323353 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.323252    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2197b3ff-694b-4b17-9916-149ec1eb9924-crio-socket\") pod \"insights-runtime-extractor-cchrc\" (UID: \"2197b3ff-694b-4b17-9916-149ec1eb9924\") " pod="openshift-insights/insights-runtime-extractor-cchrc"
Sep 11 20:01:23.323353 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.323300    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqjhc\" (UniqueName: \"kubernetes.io/projected/a9498946-2c6a-4a09-ae75-a79812f06fd2-kube-api-access-gqjhc\") pod \"dns-default-q484p\" (UID: \"a9498946-2c6a-4a09-ae75-a79812f06fd2\") " pod="openshift-dns/dns-default-q484p"
Sep 11 20:01:23.323353 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.323334    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2197b3ff-694b-4b17-9916-149ec1eb9924-data-volume\") pod \"insights-runtime-extractor-cchrc\" (UID: \"2197b3ff-694b-4b17-9916-149ec1eb9924\") " pod="openshift-insights/insights-runtime-extractor-cchrc"
Sep 11 20:01:23.323678 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.323360    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a9498946-2c6a-4a09-ae75-a79812f06fd2-config-volume\") pod \"dns-default-q484p\" (UID: \"a9498946-2c6a-4a09-ae75-a79812f06fd2\") " pod="openshift-dns/dns-default-q484p"
Sep 11 20:01:23.323678 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.323391    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txvzc\" (UniqueName: \"kubernetes.io/projected/df290aa2-b262-4940-9eea-054aca381519-kube-api-access-txvzc\") pod \"downloads-76f99584f4-wzwg7\" (UID: \"df290aa2-b262-4940-9eea-054aca381519\") " pod="openshift-console/downloads-76f99584f4-wzwg7"
Sep 11 20:01:23.323678 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.323424    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1819bd16-803f-4ec5-a141-5911232d0799-registry-tls\") pod \"image-registry-68887c9c9b-thsrj\" (UID: \"1819bd16-803f-4ec5-a141-5911232d0799\") " pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.323678 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.323447    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4qld\" (UniqueName: \"kubernetes.io/projected/def84bb5-29ef-44af-a9b8-34a1f3715012-kube-api-access-m4qld\") pod \"ingress-canary-rssdb\" (UID: \"def84bb5-29ef-44af-a9b8-34a1f3715012\") " pod="openshift-ingress-canary/ingress-canary-rssdb"
Sep 11 20:01:23.323678 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.323478    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1819bd16-803f-4ec5-a141-5911232d0799-image-registry-private-configuration\") pod \"image-registry-68887c9c9b-thsrj\" (UID: \"1819bd16-803f-4ec5-a141-5911232d0799\") " pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.323678 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.323521    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1819bd16-803f-4ec5-a141-5911232d0799-installation-pull-secrets\") pod \"image-registry-68887c9c9b-thsrj\" (UID: \"1819bd16-803f-4ec5-a141-5911232d0799\") " pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.323678 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.323544    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9498946-2c6a-4a09-ae75-a79812f06fd2-tmp-dir\") pod \"dns-default-q484p\" (UID: \"a9498946-2c6a-4a09-ae75-a79812f06fd2\") " pod="openshift-dns/dns-default-q484p"
Sep 11 20:01:23.323678 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.323574    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1819bd16-803f-4ec5-a141-5911232d0799-bound-sa-token\") pod \"image-registry-68887c9c9b-thsrj\" (UID: \"1819bd16-803f-4ec5-a141-5911232d0799\") " pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.323678 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.323584    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2197b3ff-694b-4b17-9916-149ec1eb9924-crio-socket\") pod \"insights-runtime-extractor-cchrc\" (UID: \"2197b3ff-694b-4b17-9916-149ec1eb9924\") " pod="openshift-insights/insights-runtime-extractor-cchrc"
Sep 11 20:01:23.323678 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.323598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2197b3ff-694b-4b17-9916-149ec1eb9924-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cchrc\" (UID: \"2197b3ff-694b-4b17-9916-149ec1eb9924\") " pod="openshift-insights/insights-runtime-extractor-cchrc"
Sep 11 20:01:23.323678 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.323630    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1819bd16-803f-4ec5-a141-5911232d0799-ca-trust-extracted\") pod \"image-registry-68887c9c9b-thsrj\" (UID: \"1819bd16-803f-4ec5-a141-5911232d0799\") " pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.323678 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.323645    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9498946-2c6a-4a09-ae75-a79812f06fd2-metrics-tls\") pod \"dns-default-q484p\" (UID: \"a9498946-2c6a-4a09-ae75-a79812f06fd2\") " pod="openshift-dns/dns-default-q484p"
Sep 11 20:01:23.323678 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.323649    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2197b3ff-694b-4b17-9916-149ec1eb9924-data-volume\") pod \"insights-runtime-extractor-cchrc\" (UID: \"2197b3ff-694b-4b17-9916-149ec1eb9924\") " pod="openshift-insights/insights-runtime-extractor-cchrc"
Sep 11 20:01:23.323678 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.323680    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/def84bb5-29ef-44af-a9b8-34a1f3715012-cert\") pod \"ingress-canary-rssdb\" (UID: \"def84bb5-29ef-44af-a9b8-34a1f3715012\") " pod="openshift-ingress-canary/ingress-canary-rssdb"
Sep 11 20:01:23.324340 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.323800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1819bd16-803f-4ec5-a141-5911232d0799-registry-certificates\") pod \"image-registry-68887c9c9b-thsrj\" (UID: \"1819bd16-803f-4ec5-a141-5911232d0799\") " pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.324340 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.323840    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kf6d\" (UniqueName: \"kubernetes.io/projected/2197b3ff-694b-4b17-9916-149ec1eb9924-kube-api-access-9kf6d\") pod \"insights-runtime-extractor-cchrc\" (UID: \"2197b3ff-694b-4b17-9916-149ec1eb9924\") " pod="openshift-insights/insights-runtime-extractor-cchrc"
Sep 11 20:01:23.324340 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.323840    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2197b3ff-694b-4b17-9916-149ec1eb9924-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cchrc\" (UID: \"2197b3ff-694b-4b17-9916-149ec1eb9924\") " pod="openshift-insights/insights-runtime-extractor-cchrc"
Sep 11 20:01:23.324340 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.324048    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klvb7\" (UniqueName: \"kubernetes.io/projected/1819bd16-803f-4ec5-a141-5911232d0799-kube-api-access-klvb7\") pod \"image-registry-68887c9c9b-thsrj\" (UID: \"1819bd16-803f-4ec5-a141-5911232d0799\") " pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.324340 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.324179    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1819bd16-803f-4ec5-a141-5911232d0799-trusted-ca\") pod \"image-registry-68887c9c9b-thsrj\" (UID: \"1819bd16-803f-4ec5-a141-5911232d0799\") " pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.324874 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.324853    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1819bd16-803f-4ec5-a141-5911232d0799-registry-certificates\") pod \"image-registry-68887c9c9b-thsrj\" (UID: \"1819bd16-803f-4ec5-a141-5911232d0799\") " pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.325890 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.325870    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1819bd16-803f-4ec5-a141-5911232d0799-installation-pull-secrets\") pod \"image-registry-68887c9c9b-thsrj\" (UID: \"1819bd16-803f-4ec5-a141-5911232d0799\") " pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.326011 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.325986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2197b3ff-694b-4b17-9916-149ec1eb9924-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cchrc\" (UID: \"2197b3ff-694b-4b17-9916-149ec1eb9924\") " pod="openshift-insights/insights-runtime-extractor-cchrc"
Sep 11 20:01:23.326423 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.326402    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1819bd16-803f-4ec5-a141-5911232d0799-image-registry-private-configuration\") pod \"image-registry-68887c9c9b-thsrj\" (UID: \"1819bd16-803f-4ec5-a141-5911232d0799\") " pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.326467 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.326410    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1819bd16-803f-4ec5-a141-5911232d0799-registry-tls\") pod \"image-registry-68887c9c9b-thsrj\" (UID: \"1819bd16-803f-4ec5-a141-5911232d0799\") " pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.339578 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.339559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1819bd16-803f-4ec5-a141-5911232d0799-bound-sa-token\") pod \"image-registry-68887c9c9b-thsrj\" (UID: \"1819bd16-803f-4ec5-a141-5911232d0799\") " pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.341112 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.341097    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txvzc\" (UniqueName: \"kubernetes.io/projected/df290aa2-b262-4940-9eea-054aca381519-kube-api-access-txvzc\") pod \"downloads-76f99584f4-wzwg7\" (UID: \"df290aa2-b262-4940-9eea-054aca381519\") " pod="openshift-console/downloads-76f99584f4-wzwg7"
Sep 11 20:01:23.351275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.351256    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kf6d\" (UniqueName: \"kubernetes.io/projected/2197b3ff-694b-4b17-9916-149ec1eb9924-kube-api-access-9kf6d\") pod \"insights-runtime-extractor-cchrc\" (UID: \"2197b3ff-694b-4b17-9916-149ec1eb9924\") " pod="openshift-insights/insights-runtime-extractor-cchrc"
Sep 11 20:01:23.354479 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.354457    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klvb7\" (UniqueName: \"kubernetes.io/projected/1819bd16-803f-4ec5-a141-5911232d0799-kube-api-access-klvb7\") pod \"image-registry-68887c9c9b-thsrj\" (UID: \"1819bd16-803f-4ec5-a141-5911232d0799\") " pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.358294 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.358280    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-wzwg7"
Sep 11 20:01:23.364368 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.364350    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cchrc"
Sep 11 20:01:23.370158 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.370141    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:23.425146 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.424930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a9498946-2c6a-4a09-ae75-a79812f06fd2-config-volume\") pod \"dns-default-q484p\" (UID: \"a9498946-2c6a-4a09-ae75-a79812f06fd2\") " pod="openshift-dns/dns-default-q484p"
Sep 11 20:01:23.425146 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.424984    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4qld\" (UniqueName: \"kubernetes.io/projected/def84bb5-29ef-44af-a9b8-34a1f3715012-kube-api-access-m4qld\") pod \"ingress-canary-rssdb\" (UID: \"def84bb5-29ef-44af-a9b8-34a1f3715012\") " pod="openshift-ingress-canary/ingress-canary-rssdb"
Sep 11 20:01:23.425146 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.425018    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9498946-2c6a-4a09-ae75-a79812f06fd2-tmp-dir\") pod \"dns-default-q484p\" (UID: \"a9498946-2c6a-4a09-ae75-a79812f06fd2\") " pod="openshift-dns/dns-default-q484p"
Sep 11 20:01:23.425146 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.425058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9498946-2c6a-4a09-ae75-a79812f06fd2-metrics-tls\") pod \"dns-default-q484p\" (UID: \"a9498946-2c6a-4a09-ae75-a79812f06fd2\") " pod="openshift-dns/dns-default-q484p"
Sep 11 20:01:23.425146 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.425083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/def84bb5-29ef-44af-a9b8-34a1f3715012-cert\") pod \"ingress-canary-rssdb\" (UID: \"def84bb5-29ef-44af-a9b8-34a1f3715012\") " pod="openshift-ingress-canary/ingress-canary-rssdb"
Sep 11 20:01:23.425576 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.425191    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqjhc\" (UniqueName: \"kubernetes.io/projected/a9498946-2c6a-4a09-ae75-a79812f06fd2-kube-api-access-gqjhc\") pod \"dns-default-q484p\" (UID: \"a9498946-2c6a-4a09-ae75-a79812f06fd2\") " pod="openshift-dns/dns-default-q484p"
Sep 11 20:01:23.427125 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.425725    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9498946-2c6a-4a09-ae75-a79812f06fd2-tmp-dir\") pod \"dns-default-q484p\" (UID: \"a9498946-2c6a-4a09-ae75-a79812f06fd2\") " pod="openshift-dns/dns-default-q484p"
Sep 11 20:01:23.427125 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.426303    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a9498946-2c6a-4a09-ae75-a79812f06fd2-config-volume\") pod \"dns-default-q484p\" (UID: \"a9498946-2c6a-4a09-ae75-a79812f06fd2\") " pod="openshift-dns/dns-default-q484p"
Sep 11 20:01:23.429087 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.429044    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9498946-2c6a-4a09-ae75-a79812f06fd2-metrics-tls\") pod \"dns-default-q484p\" (UID: \"a9498946-2c6a-4a09-ae75-a79812f06fd2\") " pod="openshift-dns/dns-default-q484p"
Sep 11 20:01:23.443054 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.442936    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/def84bb5-29ef-44af-a9b8-34a1f3715012-cert\") pod \"ingress-canary-rssdb\" (UID: \"def84bb5-29ef-44af-a9b8-34a1f3715012\") " pod="openshift-ingress-canary/ingress-canary-rssdb"
Sep 11 20:01:23.443577 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.443293    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4qld\" (UniqueName: \"kubernetes.io/projected/def84bb5-29ef-44af-a9b8-34a1f3715012-kube-api-access-m4qld\") pod \"ingress-canary-rssdb\" (UID: \"def84bb5-29ef-44af-a9b8-34a1f3715012\") " pod="openshift-ingress-canary/ingress-canary-rssdb"
Sep 11 20:01:23.443758 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.443740    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqjhc\" (UniqueName: \"kubernetes.io/projected/a9498946-2c6a-4a09-ae75-a79812f06fd2-kube-api-access-gqjhc\") pod \"dns-default-q484p\" (UID: \"a9498946-2c6a-4a09-ae75-a79812f06fd2\") " pod="openshift-dns/dns-default-q484p"
Sep 11 20:01:23.462213 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.462193    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q484p"
Sep 11 20:01:23.469340 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.469295    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rssdb"
Sep 11 20:01:23.645451 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.645418    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-wzwg7"]
Sep 11 20:01:23.649430 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.649406    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cchrc"]
Sep 11 20:01:23.649621 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:01:23.649595    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf290aa2_b262_4940_9eea_054aca381519.slice/crio-67796dbf9f5a19b4ef048b4510df0663f5da177e4f83c4d28a696c966d0521b7 WatchSource:0}: Error finding container 67796dbf9f5a19b4ef048b4510df0663f5da177e4f83c4d28a696c966d0521b7: Status 404 returned error can't find the container with id 67796dbf9f5a19b4ef048b4510df0663f5da177e4f83c4d28a696c966d0521b7
Sep 11 20:01:23.651125 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.651101    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68887c9c9b-thsrj"]
Sep 11 20:01:23.652749 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:01:23.652727    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2197b3ff_694b_4b17_9916_149ec1eb9924.slice/crio-1c06987583cdec93eb110662949543acb3a7a1caddb2d4cfbadcf80db93509b5 WatchSource:0}: Error finding container 1c06987583cdec93eb110662949543acb3a7a1caddb2d4cfbadcf80db93509b5: Status 404 returned error can't find the container with id 1c06987583cdec93eb110662949543acb3a7a1caddb2d4cfbadcf80db93509b5
Sep 11 20:01:23.654072 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:01:23.654049    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1819bd16_803f_4ec5_a141_5911232d0799.slice/crio-7aace77cdb6826d9c3b822b827124b0281a55b8dbbd98d1900977957c297782a WatchSource:0}: Error finding container 7aace77cdb6826d9c3b822b827124b0281a55b8dbbd98d1900977957c297782a: Status 404 returned error can't find the container with id 7aace77cdb6826d9c3b822b827124b0281a55b8dbbd98d1900977957c297782a
Sep 11 20:01:23.702866 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.702839    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68887c9c9b-thsrj" event={"ID":"1819bd16-803f-4ec5-a141-5911232d0799","Type":"ContainerStarted","Data":"7aace77cdb6826d9c3b822b827124b0281a55b8dbbd98d1900977957c297782a"}
Sep 11 20:01:23.703997 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.703975    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-wzwg7" event={"ID":"df290aa2-b262-4940-9eea-054aca381519","Type":"ContainerStarted","Data":"67796dbf9f5a19b4ef048b4510df0663f5da177e4f83c4d28a696c966d0521b7"}
Sep 11 20:01:23.705068 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.705046    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cchrc" event={"ID":"2197b3ff-694b-4b17-9916-149ec1eb9924","Type":"ContainerStarted","Data":"1c06987583cdec93eb110662949543acb3a7a1caddb2d4cfbadcf80db93509b5"}
Sep 11 20:01:23.705156 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.705075    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.710790 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.710768    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:23.744851 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.744825    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rssdb"]
Sep 11 20:01:23.745698 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.745671    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q484p"]
Sep 11 20:01:23.748364 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:01:23.748334    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef84bb5_29ef_44af_a9b8_34a1f3715012.slice/crio-3b661c51d53d5a3dfb4198b4f6ebdce16dc7870c72f46cf1777c6a960a22525d WatchSource:0}: Error finding container 3b661c51d53d5a3dfb4198b4f6ebdce16dc7870c72f46cf1777c6a960a22525d: Status 404 returned error can't find the container with id 3b661c51d53d5a3dfb4198b4f6ebdce16dc7870c72f46cf1777c6a960a22525d
Sep 11 20:01:23.749632 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:01:23.749611    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9498946_2c6a_4a09_ae75_a79812f06fd2.slice/crio-bc90fca7be4817c6cbdf58ef0dd943a95177e77e052764df2e6a82a253fe10cf WatchSource:0}: Error finding container bc90fca7be4817c6cbdf58ef0dd943a95177e77e052764df2e6a82a253fe10cf: Status 404 returned error can't find the container with id bc90fca7be4817c6cbdf58ef0dd943a95177e77e052764df2e6a82a253fe10cf
Sep 11 20:01:23.827947 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.827872    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b755347-8d4f-42bb-be5b-fff48cb4ac46-bound-sa-token\") pod \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") "
Sep 11 20:01:23.827947 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.827918    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b755347-8d4f-42bb-be5b-fff48cb4ac46-registry-certificates\") pod \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") "
Sep 11 20:01:23.828202 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.827968    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b755347-8d4f-42bb-be5b-fff48cb4ac46-image-registry-private-configuration\") pod \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") "
Sep 11 20:01:23.828202 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.828002    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b755347-8d4f-42bb-be5b-fff48cb4ac46-installation-pull-secrets\") pod \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") "
Sep 11 20:01:23.828202 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.828053    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b755347-8d4f-42bb-be5b-fff48cb4ac46-ca-trust-extracted\") pod \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") "
Sep 11 20:01:23.828202 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.828087    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b755347-8d4f-42bb-be5b-fff48cb4ac46-registry-tls\") pod \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") "
Sep 11 20:01:23.828202 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.828110    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gshlb\" (UniqueName: \"kubernetes.io/projected/6b755347-8d4f-42bb-be5b-fff48cb4ac46-kube-api-access-gshlb\") pod \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") "
Sep 11 20:01:23.828202 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.828146    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b755347-8d4f-42bb-be5b-fff48cb4ac46-trusted-ca\") pod \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\" (UID: \"6b755347-8d4f-42bb-be5b-fff48cb4ac46\") "
Sep 11 20:01:23.828517 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.828365    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b755347-8d4f-42bb-be5b-fff48cb4ac46-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6b755347-8d4f-42bb-be5b-fff48cb4ac46" (UID: "6b755347-8d4f-42bb-be5b-fff48cb4ac46"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:01:23.828743 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.828678    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b755347-8d4f-42bb-be5b-fff48cb4ac46-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6b755347-8d4f-42bb-be5b-fff48cb4ac46" (UID: "6b755347-8d4f-42bb-be5b-fff48cb4ac46"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:01:23.828872 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.828774    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b755347-8d4f-42bb-be5b-fff48cb4ac46-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6b755347-8d4f-42bb-be5b-fff48cb4ac46" (UID: "6b755347-8d4f-42bb-be5b-fff48cb4ac46"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:01:23.830572 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.830546    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b755347-8d4f-42bb-be5b-fff48cb4ac46-kube-api-access-gshlb" (OuterVolumeSpecName: "kube-api-access-gshlb") pod "6b755347-8d4f-42bb-be5b-fff48cb4ac46" (UID: "6b755347-8d4f-42bb-be5b-fff48cb4ac46"). InnerVolumeSpecName "kube-api-access-gshlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:01:23.830703 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.830679    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b755347-8d4f-42bb-be5b-fff48cb4ac46-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6b755347-8d4f-42bb-be5b-fff48cb4ac46" (UID: "6b755347-8d4f-42bb-be5b-fff48cb4ac46"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:01:23.830703 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.830691    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b755347-8d4f-42bb-be5b-fff48cb4ac46-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6b755347-8d4f-42bb-be5b-fff48cb4ac46" (UID: "6b755347-8d4f-42bb-be5b-fff48cb4ac46"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:01:23.830789 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.830737    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b755347-8d4f-42bb-be5b-fff48cb4ac46-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6b755347-8d4f-42bb-be5b-fff48cb4ac46" (UID: "6b755347-8d4f-42bb-be5b-fff48cb4ac46"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:01:23.830911 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.830891    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b755347-8d4f-42bb-be5b-fff48cb4ac46-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6b755347-8d4f-42bb-be5b-fff48cb4ac46" (UID: "6b755347-8d4f-42bb-be5b-fff48cb4ac46"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:01:23.929444 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.929412    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b755347-8d4f-42bb-be5b-fff48cb4ac46-registry-tls\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:01:23.929444 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.929437    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gshlb\" (UniqueName: \"kubernetes.io/projected/6b755347-8d4f-42bb-be5b-fff48cb4ac46-kube-api-access-gshlb\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:01:23.929444 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.929447    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b755347-8d4f-42bb-be5b-fff48cb4ac46-trusted-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:01:23.929649 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.929456    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b755347-8d4f-42bb-be5b-fff48cb4ac46-bound-sa-token\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:01:23.929649 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.929464    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b755347-8d4f-42bb-be5b-fff48cb4ac46-registry-certificates\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:01:23.929649 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.929473    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b755347-8d4f-42bb-be5b-fff48cb4ac46-image-registry-private-configuration\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:01:23.929649 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.929482    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b755347-8d4f-42bb-be5b-fff48cb4ac46-installation-pull-secrets\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:01:23.929649 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:23.929505    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b755347-8d4f-42bb-be5b-fff48cb4ac46-ca-trust-extracted\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:01:24.709783 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:24.709704    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q484p" event={"ID":"a9498946-2c6a-4a09-ae75-a79812f06fd2","Type":"ContainerStarted","Data":"bc90fca7be4817c6cbdf58ef0dd943a95177e77e052764df2e6a82a253fe10cf"}
Sep 11 20:01:24.713052 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:24.713024    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68887c9c9b-thsrj" event={"ID":"1819bd16-803f-4ec5-a141-5911232d0799","Type":"ContainerStarted","Data":"6fb30e4d51cab711b81073a7ec68e8a5432bb008bd5d3726cebfd0cba6c0db9b"}
Sep 11 20:01:24.713709 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:24.713690    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:24.716017 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:24.715915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cchrc" event={"ID":"2197b3ff-694b-4b17-9916-149ec1eb9924","Type":"ContainerStarted","Data":"c1b2d61604559f0b374b9cd87e5ecead6f3191c3b05b27682f8c39e2bd4fe651"}
Sep 11 20:01:24.716017 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:24.715967    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cchrc" event={"ID":"2197b3ff-694b-4b17-9916-149ec1eb9924","Type":"ContainerStarted","Data":"48c2485effe524159d0c103c52a13b1b19f6d56e41f1b88238746a7a02e880ea"}
Sep 11 20:01:24.723555 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:24.719156    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rssdb" event={"ID":"def84bb5-29ef-44af-a9b8-34a1f3715012","Type":"ContainerStarted","Data":"3b661c51d53d5a3dfb4198b4f6ebdce16dc7870c72f46cf1777c6a960a22525d"}
Sep 11 20:01:24.723555 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:24.719169    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78b79fd667-jpnm6"
Sep 11 20:01:24.737228 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:24.736644    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-68887c9c9b-thsrj" podStartSLOduration=1.736625246 podStartE2EDuration="1.736625246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:24.734863724 +0000 UTC m=+54.727025995" watchObservedRunningTime="2026-09-11 20:01:24.736625246 +0000 UTC m=+54.728787519"
Sep 11 20:01:24.764149 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:24.764067    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78b79fd667-jpnm6"]
Sep 11 20:01:24.767275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:24.767254    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-78b79fd667-jpnm6"]
Sep 11 20:01:26.440318 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:26.440283    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b755347-8d4f-42bb-be5b-fff48cb4ac46" path="/var/lib/kubelet/pods/6b755347-8d4f-42bb-be5b-fff48cb4ac46/volumes"
Sep 11 20:01:27.731112 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:27.731030    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cchrc" event={"ID":"2197b3ff-694b-4b17-9916-149ec1eb9924","Type":"ContainerStarted","Data":"a5e276e79b10416e66e4562b03166632ab6fc8ba6dc81b077b38a10f3ae4e0a7"}
Sep 11 20:01:27.732844 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:27.732815    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rssdb" event={"ID":"def84bb5-29ef-44af-a9b8-34a1f3715012","Type":"ContainerStarted","Data":"7380537f15b307ab961f6e0b6f4381b2c9977b6190b1f1ac9a1643463471330f"}
Sep 11 20:01:27.734860 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:27.734826    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q484p" event={"ID":"a9498946-2c6a-4a09-ae75-a79812f06fd2","Type":"ContainerStarted","Data":"acc30de1ea265c7fc750bfd668af7966eee53d928fb0d890138a1f3996230f0b"}
Sep 11 20:01:27.734990 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:27.734870    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q484p" event={"ID":"a9498946-2c6a-4a09-ae75-a79812f06fd2","Type":"ContainerStarted","Data":"29bdcf955fffbac29b9d5748d6fb5ab768d4880361721512208ee60b700be5de"}
Sep 11 20:01:27.734990 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:27.734909    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q484p"
Sep 11 20:01:27.753808 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:27.753732    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cchrc" podStartSLOduration=1.132393899 podStartE2EDuration="4.753718398s" totalImagesPullingTime="3.621324499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:01:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:27.75244854 +0000 UTC m=+57.744610809" watchObservedRunningTime="2026-09-11 20:01:27.753718398 +0000 UTC m=+57.745880659"
Sep 11 20:01:27.770207 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:27.770027    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rssdb" podStartSLOduration=1.108940403 podStartE2EDuration="4.770011592s" totalImagesPullingTime="3.661071189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:27.769802883 +0000 UTC m=+57.761965153" watchObservedRunningTime="2026-09-11 20:01:27.770011592 +0000 UTC m=+57.762173865"
Sep 11 20:01:27.792167 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:27.792116    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-q484p" podStartSLOduration=1.13173881 podStartE2EDuration="4.792099077s" totalImagesPullingTime="3.660360267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:27.791364192 +0000 UTC m=+57.783526467" watchObservedRunningTime="2026-09-11 20:01:27.792099077 +0000 UTC m=+57.784261351"
Sep 11 20:01:29.651040 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:29.650996    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j7pjf"
Sep 11 20:01:33.046912 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.046878    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56dc696459-kgzlz"]
Sep 11 20:01:33.050107 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.050080    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:01:33.055966 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.055939    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 20:01:33.056103 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.056009    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 20:01:33.056103 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.055939    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 20:01:33.056260 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.056244    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 20:01:33.056718 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.056701    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 20:01:33.057476 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.057429    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-g7fth\""
Sep 11 20:01:33.061285 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.061268    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 20:01:33.064200 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.064170    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56dc696459-kgzlz"]
Sep 11 20:01:33.099702 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.099673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d55865b-db55-4e36-ad84-52b91e019f84-oauth-serving-cert\") pod \"console-56dc696459-kgzlz\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") " pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:01:33.099810 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.099712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d55865b-db55-4e36-ad84-52b91e019f84-service-ca\") pod \"console-56dc696459-kgzlz\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") " pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:01:33.099810 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.099750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d55865b-db55-4e36-ad84-52b91e019f84-trusted-ca-bundle\") pod \"console-56dc696459-kgzlz\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") " pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:01:33.099919 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.099817    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d55865b-db55-4e36-ad84-52b91e019f84-console-oauth-config\") pod \"console-56dc696459-kgzlz\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") " pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:01:33.099919 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.099897    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d55865b-db55-4e36-ad84-52b91e019f84-console-serving-cert\") pod \"console-56dc696459-kgzlz\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") " pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:01:33.099998 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.099942    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d55865b-db55-4e36-ad84-52b91e019f84-console-config\") pod \"console-56dc696459-kgzlz\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") " pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:01:33.100048 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.099996    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqwmc\" (UniqueName: \"kubernetes.io/projected/8d55865b-db55-4e36-ad84-52b91e019f84-kube-api-access-mqwmc\") pod \"console-56dc696459-kgzlz\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") " pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:01:33.201011 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.200974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d55865b-db55-4e36-ad84-52b91e019f84-console-oauth-config\") pod \"console-56dc696459-kgzlz\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") " pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:01:33.201165 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.201039    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d55865b-db55-4e36-ad84-52b91e019f84-console-serving-cert\") pod \"console-56dc696459-kgzlz\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") " pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:01:33.201165 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.201101    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d55865b-db55-4e36-ad84-52b91e019f84-console-config\") pod \"console-56dc696459-kgzlz\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") " pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:01:33.201271 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.201247    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqwmc\" (UniqueName: \"kubernetes.io/projected/8d55865b-db55-4e36-ad84-52b91e019f84-kube-api-access-mqwmc\") pod \"console-56dc696459-kgzlz\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") " pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:01:33.201328 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.201282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d55865b-db55-4e36-ad84-52b91e019f84-oauth-serving-cert\") pod \"console-56dc696459-kgzlz\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") " pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:01:33.201328 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.201308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d55865b-db55-4e36-ad84-52b91e019f84-service-ca\") pod \"console-56dc696459-kgzlz\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") " pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:01:33.201633 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.201539    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d55865b-db55-4e36-ad84-52b91e019f84-trusted-ca-bundle\") pod \"console-56dc696459-kgzlz\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") " pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:01:33.202225 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.202171    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d55865b-db55-4e36-ad84-52b91e019f84-console-config\") pod \"console-56dc696459-kgzlz\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") " pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:01:33.202225 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.202191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d55865b-db55-4e36-ad84-52b91e019f84-service-ca\") pod \"console-56dc696459-kgzlz\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") " pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:01:33.202225 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.202218    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d55865b-db55-4e36-ad84-52b91e019f84-oauth-serving-cert\") pod \"console-56dc696459-kgzlz\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") " pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:01:33.202846 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.202806    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d55865b-db55-4e36-ad84-52b91e019f84-trusted-ca-bundle\") pod \"console-56dc696459-kgzlz\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") " pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:01:33.203956 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.203933    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d55865b-db55-4e36-ad84-52b91e019f84-console-oauth-config\") pod \"console-56dc696459-kgzlz\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") " pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:01:33.204143 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.204121    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d55865b-db55-4e36-ad84-52b91e019f84-console-serving-cert\") pod \"console-56dc696459-kgzlz\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") " pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:01:33.210947 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.210924    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqwmc\" (UniqueName: \"kubernetes.io/projected/8d55865b-db55-4e36-ad84-52b91e019f84-kube-api-access-mqwmc\") pod \"console-56dc696459-kgzlz\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") " pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:01:33.360347 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.360275    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:01:33.642594 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.640659    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56dc696459-kgzlz"]
Sep 11 20:01:33.643005 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:01:33.642972    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d55865b_db55_4e36_ad84_52b91e019f84.slice/crio-7d86b68a05a51189c23d9d9959af58a51ff4b99bf4c5b5ea7cec164562de2e78 WatchSource:0}: Error finding container 7d86b68a05a51189c23d9d9959af58a51ff4b99bf4c5b5ea7cec164562de2e78: Status 404 returned error can't find the container with id 7d86b68a05a51189c23d9d9959af58a51ff4b99bf4c5b5ea7cec164562de2e78
Sep 11 20:01:33.757009 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:33.756950    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56dc696459-kgzlz" event={"ID":"8d55865b-db55-4e36-ad84-52b91e019f84","Type":"ContainerStarted","Data":"7d86b68a05a51189c23d9d9959af58a51ff4b99bf4c5b5ea7cec164562de2e78"}
Sep 11 20:01:36.127103 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.127067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2936adde-ce1b-4859-858d-db3584a92609-original-pull-secret\") pod \"global-pull-secret-syncer-95wcq\" (UID: \"2936adde-ce1b-4859-858d-db3584a92609\") " pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:01:36.127542 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.127126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda12783-5f3e-4a3e-9fde-4681797a7b60-metrics-certs\") pod \"network-metrics-daemon-fp6qw\" (UID: \"dda12783-5f3e-4a3e-9fde-4681797a7b60\") " pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:01:36.129930 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.129901    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 20:01:36.130833 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.130802    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 20:01:36.140281 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.140229    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dda12783-5f3e-4a3e-9fde-4681797a7b60-metrics-certs\") pod \"network-metrics-daemon-fp6qw\" (UID: \"dda12783-5f3e-4a3e-9fde-4681797a7b60\") " pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:01:36.140281 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.140240    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2936adde-ce1b-4859-858d-db3584a92609-original-pull-secret\") pod \"global-pull-secret-syncer-95wcq\" (UID: \"2936adde-ce1b-4859-858d-db3584a92609\") " pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:01:36.227701 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.227662    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v2nl\" (UniqueName: \"kubernetes.io/projected/0a30ff2a-a013-4153-8a18-7a8aa6ec98f0-kube-api-access-9v2nl\") pod \"network-check-target-mrmpn\" (UID: \"0a30ff2a-a013-4153-8a18-7a8aa6ec98f0\") " pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:01:36.230522 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.230482    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 20:01:36.241425 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.241403    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 20:01:36.251449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.251427    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v2nl\" (UniqueName: \"kubernetes.io/projected/0a30ff2a-a013-4153-8a18-7a8aa6ec98f0-kube-api-access-9v2nl\") pod \"network-check-target-mrmpn\" (UID: \"0a30ff2a-a013-4153-8a18-7a8aa6ec98f0\") " pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:01:36.345710 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.345667    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95wcq"
Sep 11 20:01:36.353605 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.353582    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b7zd7\""
Sep 11 20:01:36.357764 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.357744    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fv8gc\""
Sep 11 20:01:36.361744 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.361727    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fp6qw"
Sep 11 20:01:36.366438 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.366408    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:01:36.843098 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.843063    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-tcl2z"]
Sep 11 20:01:36.845463 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.845441    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tcl2z"
Sep 11 20:01:36.848174 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.848151    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 11 20:01:36.848517 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.848323    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 20:01:36.848517 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.848428    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 11 20:01:36.848517 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.848478    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 20:01:36.849279 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.849257    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 20:01:36.849383 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.849260    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-ll6lr\""
Sep 11 20:01:36.852632 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.852611    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lw7xn"]
Sep 11 20:01:36.855000 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.854980    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:36.857454 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.857438    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 20:01:36.857610 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.857460    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6rwxf\""
Sep 11 20:01:36.857610 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.857440    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 20:01:36.857812 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.857792    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 20:01:36.863855 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.863836    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-tcl2z"]
Sep 11 20:01:36.867034 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.867013    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq"]
Sep 11 20:01:36.869397 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.869275    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq"
Sep 11 20:01:36.890520 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.873874    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 11 20:01:36.890520 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.874185    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 11 20:01:36.890520 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.874389    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-7l7k9\""
Sep 11 20:01:36.890520 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.874744    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 11 20:01:36.893391 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.893372    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq"]
Sep 11 20:01:36.933746 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.933712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/28df1910-4b5d-486d-b41e-faad23a61df6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-tcl2z\" (UID: \"28df1910-4b5d-486d-b41e-faad23a61df6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tcl2z"
Sep 11 20:01:36.933913 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.933760    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8dad13e0-28a0-4906-bc88-53f2e6c8f298-node-exporter-tls\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:36.933913 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.933789    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3146fb02-8e77-4589-b856-eac2cecf5326-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-4n8lq\" (UID: \"3146fb02-8e77-4589-b856-eac2cecf5326\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq"
Sep 11 20:01:36.933913 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.933818    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28df1910-4b5d-486d-b41e-faad23a61df6-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-tcl2z\" (UID: \"28df1910-4b5d-486d-b41e-faad23a61df6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tcl2z"
Sep 11 20:01:36.933913 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.933838    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3146fb02-8e77-4589-b856-eac2cecf5326-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-4n8lq\" (UID: \"3146fb02-8e77-4589-b856-eac2cecf5326\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq"
Sep 11 20:01:36.933913 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.933855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sxtl\" (UniqueName: \"kubernetes.io/projected/3146fb02-8e77-4589-b856-eac2cecf5326-kube-api-access-4sxtl\") pod \"kube-state-metrics-59fbdd77d-4n8lq\" (UID: \"3146fb02-8e77-4589-b856-eac2cecf5326\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq"
Sep 11 20:01:36.933913 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.933879    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8dad13e0-28a0-4906-bc88-53f2e6c8f298-root\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:36.933913 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.933903    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3146fb02-8e77-4589-b856-eac2cecf5326-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-4n8lq\" (UID: \"3146fb02-8e77-4589-b856-eac2cecf5326\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq"
Sep 11 20:01:36.934234 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.933935    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nphd7\" (UniqueName: \"kubernetes.io/projected/8dad13e0-28a0-4906-bc88-53f2e6c8f298-kube-api-access-nphd7\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:36.934234 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.933959    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z86d9\" (UniqueName: \"kubernetes.io/projected/28df1910-4b5d-486d-b41e-faad23a61df6-kube-api-access-z86d9\") pod \"openshift-state-metrics-67dcf8c979-tcl2z\" (UID: \"28df1910-4b5d-486d-b41e-faad23a61df6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tcl2z"
Sep 11 20:01:36.934234 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.933986    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8dad13e0-28a0-4906-bc88-53f2e6c8f298-metrics-client-ca\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:36.934234 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.934033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8dad13e0-28a0-4906-bc88-53f2e6c8f298-node-exporter-textfile\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:36.934234 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.934076    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8dad13e0-28a0-4906-bc88-53f2e6c8f298-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:36.934234 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.934105    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3146fb02-8e77-4589-b856-eac2cecf5326-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-4n8lq\" (UID: \"3146fb02-8e77-4589-b856-eac2cecf5326\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq"
Sep 11 20:01:36.934234 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.934131    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3146fb02-8e77-4589-b856-eac2cecf5326-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-4n8lq\" (UID: \"3146fb02-8e77-4589-b856-eac2cecf5326\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq"
Sep 11 20:01:36.934234 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.934162    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8dad13e0-28a0-4906-bc88-53f2e6c8f298-node-exporter-accelerators-collector-config\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:36.934234 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.934208    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8dad13e0-28a0-4906-bc88-53f2e6c8f298-sys\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:36.934568 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.934246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28df1910-4b5d-486d-b41e-faad23a61df6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-tcl2z\" (UID: \"28df1910-4b5d-486d-b41e-faad23a61df6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tcl2z"
Sep 11 20:01:36.934568 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:36.934266    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8dad13e0-28a0-4906-bc88-53f2e6c8f298-node-exporter-wtmp\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:37.035120 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.035092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z86d9\" (UniqueName: \"kubernetes.io/projected/28df1910-4b5d-486d-b41e-faad23a61df6-kube-api-access-z86d9\") pod \"openshift-state-metrics-67dcf8c979-tcl2z\" (UID: \"28df1910-4b5d-486d-b41e-faad23a61df6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tcl2z"
Sep 11 20:01:37.035303 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.035126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8dad13e0-28a0-4906-bc88-53f2e6c8f298-metrics-client-ca\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:37.035303 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.035146    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8dad13e0-28a0-4906-bc88-53f2e6c8f298-node-exporter-textfile\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:37.035303 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.035272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8dad13e0-28a0-4906-bc88-53f2e6c8f298-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:37.035465 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.035311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3146fb02-8e77-4589-b856-eac2cecf5326-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-4n8lq\" (UID: \"3146fb02-8e77-4589-b856-eac2cecf5326\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq"
Sep 11 20:01:37.035549 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.035460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3146fb02-8e77-4589-b856-eac2cecf5326-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-4n8lq\" (UID: \"3146fb02-8e77-4589-b856-eac2cecf5326\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq"
Sep 11 20:01:37.035549 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.035476    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8dad13e0-28a0-4906-bc88-53f2e6c8f298-node-exporter-textfile\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:37.035656 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.035575    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8dad13e0-28a0-4906-bc88-53f2e6c8f298-node-exporter-accelerators-collector-config\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:37.035656 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.035612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8dad13e0-28a0-4906-bc88-53f2e6c8f298-sys\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:37.035656 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.035649    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28df1910-4b5d-486d-b41e-faad23a61df6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-tcl2z\" (UID: \"28df1910-4b5d-486d-b41e-faad23a61df6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tcl2z"
Sep 11 20:01:37.035831 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.035720    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8dad13e0-28a0-4906-bc88-53f2e6c8f298-sys\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:37.035831 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.035765    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8dad13e0-28a0-4906-bc88-53f2e6c8f298-node-exporter-wtmp\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:37.035831 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.035820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/28df1910-4b5d-486d-b41e-faad23a61df6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-tcl2z\" (UID: \"28df1910-4b5d-486d-b41e-faad23a61df6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tcl2z"
Sep 11 20:01:37.035981 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.035832    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8dad13e0-28a0-4906-bc88-53f2e6c8f298-metrics-client-ca\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:37.035981 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.035852    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8dad13e0-28a0-4906-bc88-53f2e6c8f298-node-exporter-tls\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:37.035981 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.035879    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3146fb02-8e77-4589-b856-eac2cecf5326-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-4n8lq\" (UID: \"3146fb02-8e77-4589-b856-eac2cecf5326\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq"
Sep 11 20:01:37.035981 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.035909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28df1910-4b5d-486d-b41e-faad23a61df6-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-tcl2z\" (UID: \"28df1910-4b5d-486d-b41e-faad23a61df6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tcl2z"
Sep 11 20:01:37.035981 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.035937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3146fb02-8e77-4589-b856-eac2cecf5326-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-4n8lq\" (UID: \"3146fb02-8e77-4589-b856-eac2cecf5326\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq"
Sep 11 20:01:37.035981 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.035969    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sxtl\" (UniqueName: \"kubernetes.io/projected/3146fb02-8e77-4589-b856-eac2cecf5326-kube-api-access-4sxtl\") pod \"kube-state-metrics-59fbdd77d-4n8lq\" (UID: \"3146fb02-8e77-4589-b856-eac2cecf5326\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq"
Sep 11 20:01:37.035981 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.035971    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8dad13e0-28a0-4906-bc88-53f2e6c8f298-node-exporter-wtmp\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:37.036336 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.036011    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8dad13e0-28a0-4906-bc88-53f2e6c8f298-root\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:37.036336 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.036087    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3146fb02-8e77-4589-b856-eac2cecf5326-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-4n8lq\" (UID: \"3146fb02-8e77-4589-b856-eac2cecf5326\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq"
Sep 11 20:01:37.036336 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.036105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3146fb02-8e77-4589-b856-eac2cecf5326-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-4n8lq\" (UID: \"3146fb02-8e77-4589-b856-eac2cecf5326\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq"
Sep 11 20:01:37.036336 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.036125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nphd7\" (UniqueName: \"kubernetes.io/projected/8dad13e0-28a0-4906-bc88-53f2e6c8f298-kube-api-access-nphd7\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:37.036336 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.036168    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8dad13e0-28a0-4906-bc88-53f2e6c8f298-node-exporter-accelerators-collector-config\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:37.036336 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.036260    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3146fb02-8e77-4589-b856-eac2cecf5326-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-4n8lq\" (UID: \"3146fb02-8e77-4589-b856-eac2cecf5326\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq"
Sep 11 20:01:37.036913 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.036869    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8dad13e0-28a0-4906-bc88-53f2e6c8f298-root\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:37.037036 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.036911    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3146fb02-8e77-4589-b856-eac2cecf5326-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-4n8lq\" (UID: \"3146fb02-8e77-4589-b856-eac2cecf5326\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq"
Sep 11 20:01:37.037118 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.037096    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/28df1910-4b5d-486d-b41e-faad23a61df6-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-tcl2z\" (UID: \"28df1910-4b5d-486d-b41e-faad23a61df6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tcl2z"
Sep 11 20:01:37.043271 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.038699    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/28df1910-4b5d-486d-b41e-faad23a61df6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-tcl2z\" (UID: \"28df1910-4b5d-486d-b41e-faad23a61df6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tcl2z"
Sep 11 20:01:37.043271 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.039283    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/28df1910-4b5d-486d-b41e-faad23a61df6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-tcl2z\" (UID: \"28df1910-4b5d-486d-b41e-faad23a61df6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tcl2z"
Sep 11 20:01:37.043271 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.039373    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8dad13e0-28a0-4906-bc88-53f2e6c8f298-node-exporter-tls\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:37.043271 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.039468    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8dad13e0-28a0-4906-bc88-53f2e6c8f298-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:37.043271 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.039656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3146fb02-8e77-4589-b856-eac2cecf5326-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-4n8lq\" (UID: \"3146fb02-8e77-4589-b856-eac2cecf5326\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq"
Sep 11 20:01:37.043271 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.039710    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3146fb02-8e77-4589-b856-eac2cecf5326-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-4n8lq\" (UID: \"3146fb02-8e77-4589-b856-eac2cecf5326\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq"
Sep 11 20:01:37.044843 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.044817    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z86d9\" (UniqueName: \"kubernetes.io/projected/28df1910-4b5d-486d-b41e-faad23a61df6-kube-api-access-z86d9\") pod \"openshift-state-metrics-67dcf8c979-tcl2z\" (UID: \"28df1910-4b5d-486d-b41e-faad23a61df6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tcl2z"
Sep 11 20:01:37.045741 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.045719    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sxtl\" (UniqueName: \"kubernetes.io/projected/3146fb02-8e77-4589-b856-eac2cecf5326-kube-api-access-4sxtl\") pod \"kube-state-metrics-59fbdd77d-4n8lq\" (UID: \"3146fb02-8e77-4589-b856-eac2cecf5326\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq"
Sep 11 20:01:37.045923 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.045905    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nphd7\" (UniqueName: \"kubernetes.io/projected/8dad13e0-28a0-4906-bc88-53f2e6c8f298-kube-api-access-nphd7\") pod \"node-exporter-lw7xn\" (UID: \"8dad13e0-28a0-4906-bc88-53f2e6c8f298\") " pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:37.157125 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.157047    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tcl2z"
Sep 11 20:01:37.166006 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.165986    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lw7xn"
Sep 11 20:01:37.200826 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.200800    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq"
Sep 11 20:01:37.741106 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:37.741078    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q484p"
Sep 11 20:01:41.137988 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.137954    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-954bf79db-prgff"]
Sep 11 20:01:41.140449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.140429    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:01:41.143084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.143064    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 11 20:01:41.144236 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.144217    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 11 20:01:41.144342 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.144220    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 11 20:01:41.144408 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.144344    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xfgqf\""
Sep 11 20:01:41.144620 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.144600    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-e39575c3s8qhc\""
Sep 11 20:01:41.144800 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.144761    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 20:01:41.150405 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.150385    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-954bf79db-prgff"]
Sep 11 20:01:41.167260 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.167232    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98c6ffec-27b5-43c0-84d5-7884d6cb7099-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-954bf79db-prgff\" (UID: \"98c6ffec-27b5-43c0-84d5-7884d6cb7099\") " pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:01:41.167372 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.167276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdjtl\" (UniqueName: \"kubernetes.io/projected/98c6ffec-27b5-43c0-84d5-7884d6cb7099-kube-api-access-zdjtl\") pod \"metrics-server-954bf79db-prgff\" (UID: \"98c6ffec-27b5-43c0-84d5-7884d6cb7099\") " pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:01:41.167372 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.167309    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/98c6ffec-27b5-43c0-84d5-7884d6cb7099-audit-log\") pod \"metrics-server-954bf79db-prgff\" (UID: \"98c6ffec-27b5-43c0-84d5-7884d6cb7099\") " pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:01:41.167453 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.167394    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/98c6ffec-27b5-43c0-84d5-7884d6cb7099-secret-metrics-server-tls\") pod \"metrics-server-954bf79db-prgff\" (UID: \"98c6ffec-27b5-43c0-84d5-7884d6cb7099\") " pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:01:41.167529 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.167458    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/98c6ffec-27b5-43c0-84d5-7884d6cb7099-client-ca-bundle\") pod \"metrics-server-954bf79db-prgff\" (UID: \"98c6ffec-27b5-43c0-84d5-7884d6cb7099\") " pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:01:41.167529 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.167486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/98c6ffec-27b5-43c0-84d5-7884d6cb7099-secret-metrics-server-client-certs\") pod \"metrics-server-954bf79db-prgff\" (UID: \"98c6ffec-27b5-43c0-84d5-7884d6cb7099\") " pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:01:41.167630 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.167537    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/98c6ffec-27b5-43c0-84d5-7884d6cb7099-metrics-server-audit-profiles\") pod \"metrics-server-954bf79db-prgff\" (UID: \"98c6ffec-27b5-43c0-84d5-7884d6cb7099\") " pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:01:41.268178 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.268140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/98c6ffec-27b5-43c0-84d5-7884d6cb7099-secret-metrics-server-client-certs\") pod \"metrics-server-954bf79db-prgff\" (UID: \"98c6ffec-27b5-43c0-84d5-7884d6cb7099\") " pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:01:41.268178 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.268181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/98c6ffec-27b5-43c0-84d5-7884d6cb7099-metrics-server-audit-profiles\") pod \"metrics-server-954bf79db-prgff\" (UID: \"98c6ffec-27b5-43c0-84d5-7884d6cb7099\") " pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:01:41.268420 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.268221    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98c6ffec-27b5-43c0-84d5-7884d6cb7099-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-954bf79db-prgff\" (UID: \"98c6ffec-27b5-43c0-84d5-7884d6cb7099\") " pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:01:41.268420 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.268251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdjtl\" (UniqueName: \"kubernetes.io/projected/98c6ffec-27b5-43c0-84d5-7884d6cb7099-kube-api-access-zdjtl\") pod \"metrics-server-954bf79db-prgff\" (UID: \"98c6ffec-27b5-43c0-84d5-7884d6cb7099\") " pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:01:41.268420 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.268320    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/98c6ffec-27b5-43c0-84d5-7884d6cb7099-audit-log\") pod \"metrics-server-954bf79db-prgff\" (UID: \"98c6ffec-27b5-43c0-84d5-7884d6cb7099\") " pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:01:41.268420 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.268391    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/98c6ffec-27b5-43c0-84d5-7884d6cb7099-secret-metrics-server-tls\") pod \"metrics-server-954bf79db-prgff\" (UID: \"98c6ffec-27b5-43c0-84d5-7884d6cb7099\") " pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:01:41.268662 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.268443    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/98c6ffec-27b5-43c0-84d5-7884d6cb7099-client-ca-bundle\") pod \"metrics-server-954bf79db-prgff\" (UID: \"98c6ffec-27b5-43c0-84d5-7884d6cb7099\") " pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:01:41.268818 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.268795    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/98c6ffec-27b5-43c0-84d5-7884d6cb7099-audit-log\") pod \"metrics-server-954bf79db-prgff\" (UID: \"98c6ffec-27b5-43c0-84d5-7884d6cb7099\") " pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:01:41.269123 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.269094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98c6ffec-27b5-43c0-84d5-7884d6cb7099-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-954bf79db-prgff\" (UID: \"98c6ffec-27b5-43c0-84d5-7884d6cb7099\") " pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:01:41.269841 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.269802    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/98c6ffec-27b5-43c0-84d5-7884d6cb7099-metrics-server-audit-profiles\") pod \"metrics-server-954bf79db-prgff\" (UID: \"98c6ffec-27b5-43c0-84d5-7884d6cb7099\") " pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:01:41.271083 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.271060    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/98c6ffec-27b5-43c0-84d5-7884d6cb7099-secret-metrics-server-client-certs\") pod \"metrics-server-954bf79db-prgff\" (UID: \"98c6ffec-27b5-43c0-84d5-7884d6cb7099\") " pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:01:41.271290 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.271270    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/98c6ffec-27b5-43c0-84d5-7884d6cb7099-secret-metrics-server-tls\") pod \"metrics-server-954bf79db-prgff\" (UID: \"98c6ffec-27b5-43c0-84d5-7884d6cb7099\") " pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:01:41.271290 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.271277    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/98c6ffec-27b5-43c0-84d5-7884d6cb7099-client-ca-bundle\") pod \"metrics-server-954bf79db-prgff\" (UID: \"98c6ffec-27b5-43c0-84d5-7884d6cb7099\") " pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:01:41.276906 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.276885    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdjtl\" (UniqueName: \"kubernetes.io/projected/98c6ffec-27b5-43c0-84d5-7884d6cb7099-kube-api-access-zdjtl\") pod \"metrics-server-954bf79db-prgff\" (UID: \"98c6ffec-27b5-43c0-84d5-7884d6cb7099\") " pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:01:41.451634 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.451602    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:01:41.643455 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.643416    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-rnd5p"]
Sep 11 20:01:41.646320 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.646290    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnd5p"
Sep 11 20:01:41.649760 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.649739    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-jczwv\""
Sep 11 20:01:41.651667 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.651644    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 11 20:01:41.658342 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.658271    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-rnd5p"]
Sep 11 20:01:41.659380 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.659352    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56dc696459-kgzlz"]
Sep 11 20:01:41.671372 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.671350    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7e42b56b-595f-476f-8a6d-a407bbc00233-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-rnd5p\" (UID: \"7e42b56b-595f-476f-8a6d-a407bbc00233\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnd5p"
Sep 11 20:01:41.671470 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.671388    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvf9b\" (UniqueName: \"kubernetes.io/projected/7e42b56b-595f-476f-8a6d-a407bbc00233-kube-api-access-wvf9b\") pod \"monitoring-plugin-5d85778b9d-rnd5p\" (UID: \"7e42b56b-595f-476f-8a6d-a407bbc00233\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnd5p"
Sep 11 20:01:41.772347 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.772262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7e42b56b-595f-476f-8a6d-a407bbc00233-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-rnd5p\" (UID: \"7e42b56b-595f-476f-8a6d-a407bbc00233\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnd5p"
Sep 11 20:01:41.772347 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.772306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvf9b\" (UniqueName: \"kubernetes.io/projected/7e42b56b-595f-476f-8a6d-a407bbc00233-kube-api-access-wvf9b\") pod \"monitoring-plugin-5d85778b9d-rnd5p\" (UID: \"7e42b56b-595f-476f-8a6d-a407bbc00233\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnd5p"
Sep 11 20:01:41.774928 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.774904    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7e42b56b-595f-476f-8a6d-a407bbc00233-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-rnd5p\" (UID: \"7e42b56b-595f-476f-8a6d-a407bbc00233\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnd5p"
Sep 11 20:01:41.780395 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.780370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvf9b\" (UniqueName: \"kubernetes.io/projected/7e42b56b-595f-476f-8a6d-a407bbc00233-kube-api-access-wvf9b\") pod \"monitoring-plugin-5d85778b9d-rnd5p\" (UID: \"7e42b56b-595f-476f-8a6d-a407bbc00233\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnd5p"
Sep 11 20:01:41.957705 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:41.957663    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnd5p"
Sep 11 20:01:43.374633 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:43.374598    2580 patch_prober.go:28] interesting pod/image-registry-68887c9c9b-thsrj container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 20:01:43.375047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:43.374688    2580 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-68887c9c9b-thsrj" podUID="1819bd16-803f-4ec5-a141-5911232d0799" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 20:01:43.423924 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:01:43.423878    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dad13e0_28a0_4906_bc88_53f2e6c8f298.slice/crio-b6b04fc85a7c3bc7204095c95bf7afc2db26337743341dfd819a461a4de6b293 WatchSource:0}: Error finding container b6b04fc85a7c3bc7204095c95bf7afc2db26337743341dfd819a461a4de6b293: Status 404 returned error can't find the container with id b6b04fc85a7c3bc7204095c95bf7afc2db26337743341dfd819a461a4de6b293
Sep 11 20:01:43.707515 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:43.707471    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-rnd5p"]
Sep 11 20:01:43.709999 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:43.709976    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-tcl2z"]
Sep 11 20:01:43.711011 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:01:43.710983    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e42b56b_595f_476f_8a6d_a407bbc00233.slice/crio-eec6f7b9bac175a8a07b70aaa2f6bd5731f7bb526888c43bda3e60727bce9823 WatchSource:0}: Error finding container eec6f7b9bac175a8a07b70aaa2f6bd5731f7bb526888c43bda3e60727bce9823: Status 404 returned error can't find the container with id eec6f7b9bac175a8a07b70aaa2f6bd5731f7bb526888c43bda3e60727bce9823
Sep 11 20:01:43.714177 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:01:43.714153    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28df1910_4b5d_486d_b41e_faad23a61df6.slice/crio-c915efa884ac47d6fa18494467c39710348dc365c80acae3895b719b2263bf6d WatchSource:0}: Error finding container c915efa884ac47d6fa18494467c39710348dc365c80acae3895b719b2263bf6d: Status 404 returned error can't find the container with id c915efa884ac47d6fa18494467c39710348dc365c80acae3895b719b2263bf6d
Sep 11 20:01:43.715305 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:43.715280    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq"]
Sep 11 20:01:43.718216 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:01:43.718191    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3146fb02_8e77_4589_b856_eac2cecf5326.slice/crio-57cf3fc066a6a092436de9b21f611e503f16d7f6dff860fb07aad141ae3a948d WatchSource:0}: Error finding container 57cf3fc066a6a092436de9b21f611e503f16d7f6dff860fb07aad141ae3a948d: Status 404 returned error can't find the container with id 57cf3fc066a6a092436de9b21f611e503f16d7f6dff860fb07aad141ae3a948d
Sep 11 20:01:43.718216 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:43.718202    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-954bf79db-prgff"]
Sep 11 20:01:43.721277 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:01:43.721257    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c6ffec_27b5_43c0_84d5_7884d6cb7099.slice/crio-ee3a0a24afa190a9d55dba814935beb0bf2bea4456083b3b42c11e1a4ebb60af WatchSource:0}: Error finding container ee3a0a24afa190a9d55dba814935beb0bf2bea4456083b3b42c11e1a4ebb60af: Status 404 returned error can't find the container with id ee3a0a24afa190a9d55dba814935beb0bf2bea4456083b3b42c11e1a4ebb60af
Sep 11 20:01:43.787054 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:43.786645    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-95wcq"]
Sep 11 20:01:43.791445 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:43.791018    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mrmpn"]
Sep 11 20:01:43.791445 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:43.791287    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fp6qw"]
Sep 11 20:01:43.796809 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:43.796682    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lw7xn" event={"ID":"8dad13e0-28a0-4906-bc88-53f2e6c8f298","Type":"ContainerStarted","Data":"b6b04fc85a7c3bc7204095c95bf7afc2db26337743341dfd819a461a4de6b293"}
Sep 11 20:01:43.799050 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:43.799022    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-954bf79db-prgff" event={"ID":"98c6ffec-27b5-43c0-84d5-7884d6cb7099","Type":"ContainerStarted","Data":"ee3a0a24afa190a9d55dba814935beb0bf2bea4456083b3b42c11e1a4ebb60af"}
Sep 11 20:01:43.803110 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:01:43.801941    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a30ff2a_a013_4153_8a18_7a8aa6ec98f0.slice/crio-4e0cb740ea3e3d784d473c3ae2ea58d1f81ae6a9defcd62e6b8deffaba53dd54 WatchSource:0}: Error finding container 4e0cb740ea3e3d784d473c3ae2ea58d1f81ae6a9defcd62e6b8deffaba53dd54: Status 404 returned error can't find the container with id 4e0cb740ea3e3d784d473c3ae2ea58d1f81ae6a9defcd62e6b8deffaba53dd54
Sep 11 20:01:43.803361 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:01:43.803105    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2936adde_ce1b_4859_858d_db3584a92609.slice/crio-78ab7b0deafada99b17eb60f71258c2e40482206dfd2775d563f6d86b0fb717c WatchSource:0}: Error finding container 78ab7b0deafada99b17eb60f71258c2e40482206dfd2775d563f6d86b0fb717c: Status 404 returned error can't find the container with id 78ab7b0deafada99b17eb60f71258c2e40482206dfd2775d563f6d86b0fb717c
Sep 11 20:01:43.804137 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:01:43.804114    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda12783_5f3e_4a3e_9fde_4681797a7b60.slice/crio-29c3fa0d9893f062383005525f1dd95d4a53b336900e5d741ad8a7a2a411fac4 WatchSource:0}: Error finding container 29c3fa0d9893f062383005525f1dd95d4a53b336900e5d741ad8a7a2a411fac4: Status 404 returned error can't find the container with id 29c3fa0d9893f062383005525f1dd95d4a53b336900e5d741ad8a7a2a411fac4
Sep 11 20:01:43.804322 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:43.804294    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tcl2z" event={"ID":"28df1910-4b5d-486d-b41e-faad23a61df6","Type":"ContainerStarted","Data":"6905044c03a0eeafbe24102a0f84269ab8c46e46c6dd094ef304989b7283450e"}
Sep 11 20:01:43.804394 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:43.804355    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tcl2z" event={"ID":"28df1910-4b5d-486d-b41e-faad23a61df6","Type":"ContainerStarted","Data":"c915efa884ac47d6fa18494467c39710348dc365c80acae3895b719b2263bf6d"}
Sep 11 20:01:43.808380 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:43.808318    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq" event={"ID":"3146fb02-8e77-4589-b856-eac2cecf5326","Type":"ContainerStarted","Data":"57cf3fc066a6a092436de9b21f611e503f16d7f6dff860fb07aad141ae3a948d"}
Sep 11 20:01:43.810396 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:43.810171    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnd5p" event={"ID":"7e42b56b-595f-476f-8a6d-a407bbc00233","Type":"ContainerStarted","Data":"eec6f7b9bac175a8a07b70aaa2f6bd5731f7bb526888c43bda3e60727bce9823"}
Sep 11 20:01:43.816559 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:43.816536    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-wzwg7" event={"ID":"df290aa2-b262-4940-9eea-054aca381519","Type":"ContainerStarted","Data":"4c12a9a4398322b03189e8529cf6a5cd8a600a348d6dc0fed0a06058fbf65bf4"}
Sep 11 20:01:43.816720 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:43.816704    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-wzwg7"
Sep 11 20:01:43.820374 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:43.819570    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56dc696459-kgzlz" event={"ID":"8d55865b-db55-4e36-ad84-52b91e019f84","Type":"ContainerStarted","Data":"4b363d6634c2b2d9361177f8f5a2108b8d2b42226c8e4a2764b571f5b2f7e1bd"}
Sep 11 20:01:43.827443 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:43.827406    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-wzwg7"
Sep 11 20:01:43.835291 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:43.835248    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-wzwg7" podStartSLOduration=0.98139486 podStartE2EDuration="20.835213003s" totalImagesPullingTime="19.853818143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:43.835154341 +0000 UTC m=+73.827316611" watchObservedRunningTime="2026-09-11 20:01:43.835213003 +0000 UTC m=+73.827375270"
Sep 11 20:01:43.854556 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:43.854441    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56dc696459-kgzlz" podStartSLOduration=1.089718148 podStartE2EDuration="10.854420972s" totalImagesPullingTime="9.764702824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:43.852983057 +0000 UTC m=+73.845145329" watchObservedRunningTime="2026-09-11 20:01:43.854420972 +0000 UTC m=+73.846583241"
Sep 11 20:01:44.828099 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:44.828004    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tcl2z" event={"ID":"28df1910-4b5d-486d-b41e-faad23a61df6","Type":"ContainerStarted","Data":"87c276244c6202320f768f1da827c874b79dfe50ebef9db2754666bdf7860036"}
Sep 11 20:01:44.830541 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:44.830414    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mrmpn" event={"ID":"0a30ff2a-a013-4153-8a18-7a8aa6ec98f0","Type":"ContainerStarted","Data":"4e0cb740ea3e3d784d473c3ae2ea58d1f81ae6a9defcd62e6b8deffaba53dd54"}
Sep 11 20:01:44.832265 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:44.832185    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-95wcq" event={"ID":"2936adde-ce1b-4859-858d-db3584a92609","Type":"ContainerStarted","Data":"78ab7b0deafada99b17eb60f71258c2e40482206dfd2775d563f6d86b0fb717c"}
Sep 11 20:01:44.834922 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:44.834859    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fp6qw" event={"ID":"dda12783-5f3e-4a3e-9fde-4681797a7b60","Type":"ContainerStarted","Data":"29c3fa0d9893f062383005525f1dd95d4a53b336900e5d741ad8a7a2a411fac4"}
Sep 11 20:01:45.846053 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:45.844914    2580 generic.go:356] "Generic (PLEG): container finished" podID="8dad13e0-28a0-4906-bc88-53f2e6c8f298" containerID="b4fb805150999cf3b1f84d415524bb34cecf20e66f057b2fc4ec121a055d728b" exitCode=0
Sep 11 20:01:45.847045 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:45.846165    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lw7xn" event={"ID":"8dad13e0-28a0-4906-bc88-53f2e6c8f298","Type":"ContainerDied","Data":"b4fb805150999cf3b1f84d415524bb34cecf20e66f057b2fc4ec121a055d728b"}
Sep 11 20:01:46.732520 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:46.732437    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-68887c9c9b-thsrj"
Sep 11 20:01:52.881159 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:52.881121    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fp6qw" event={"ID":"dda12783-5f3e-4a3e-9fde-4681797a7b60","Type":"ContainerStarted","Data":"1ebe83912796858b6d3afdf0f69c41c95108370ba721f98ef5884f80e787c655"}
Sep 11 20:01:52.883926 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:52.883896    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lw7xn" event={"ID":"8dad13e0-28a0-4906-bc88-53f2e6c8f298","Type":"ContainerStarted","Data":"c2b3876d2dcbf993c427c347bccd276425b344e89fef7af4bb17c0eca69aba4a"}
Sep 11 20:01:52.886393 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:52.886257    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-954bf79db-prgff" event={"ID":"98c6ffec-27b5-43c0-84d5-7884d6cb7099","Type":"ContainerStarted","Data":"f45003f095b9977d2688e8a594fcc901a9ff3a8135038106470e24e254696c56"}
Sep 11 20:01:52.888901 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:52.888868    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq" event={"ID":"3146fb02-8e77-4589-b856-eac2cecf5326","Type":"ContainerStarted","Data":"2ed8b307ba5f217ef4efecc4ab29e1ba75143e8df26da46be392928bfc4ca185"}
Sep 11 20:01:52.891542 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:52.891454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnd5p" event={"ID":"7e42b56b-595f-476f-8a6d-a407bbc00233","Type":"ContainerStarted","Data":"eaba769a81504f94ee40bf71c96560b7b4eaf0748d0a7b3a194596c36ab91167"}
Sep 11 20:01:52.892038 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:52.891992    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnd5p"
Sep 11 20:01:52.900172 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:52.900148    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnd5p"
Sep 11 20:01:52.907607 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:52.907560    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-954bf79db-prgff" podStartSLOduration=2.955115983 podStartE2EDuration="11.907543738s" totalImagesPullingTime="8.952427755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:52.905145895 +0000 UTC m=+82.897308167" watchObservedRunningTime="2026-09-11 20:01:52.907543738 +0000 UTC m=+82.899706008"
Sep 11 20:01:52.931257 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:52.930273    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rnd5p" podStartSLOduration=2.968036525 podStartE2EDuration="11.930252772s" totalImagesPullingTime="8.962216247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:52.929030591 +0000 UTC m=+82.921192862" watchObservedRunningTime="2026-09-11 20:01:52.930252772 +0000 UTC m=+82.922415044"
Sep 11 20:01:53.360924 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:53.360887    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:01:53.898164 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:53.898128    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lw7xn" event={"ID":"8dad13e0-28a0-4906-bc88-53f2e6c8f298","Type":"ContainerStarted","Data":"1464a2c586c67de02ad6af80791313e14917966ba04adfd86cc4fbe0073d4752"}
Sep 11 20:01:53.900420 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:53.900396    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tcl2z" event={"ID":"28df1910-4b5d-486d-b41e-faad23a61df6","Type":"ContainerStarted","Data":"57d970ae19fe88352210ecd958491f93c1bc5b966d4d4d91672b16060927688f"}
Sep 11 20:01:53.902845 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:53.902820    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq" event={"ID":"3146fb02-8e77-4589-b856-eac2cecf5326","Type":"ContainerStarted","Data":"d837c128f7088b68d95a691a70c55cdbc9a03690f2b7f6fcbd3b7b6f2cc1bb34"}
Sep 11 20:01:53.902977 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:53.902875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq" event={"ID":"3146fb02-8e77-4589-b856-eac2cecf5326","Type":"ContainerStarted","Data":"897f00b6e09d7bcdf17ae6fa56edc5f8d74942becca6b37dbdc8189d584d2f09"}
Sep 11 20:01:53.904527 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:53.904502    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mrmpn" event={"ID":"0a30ff2a-a013-4153-8a18-7a8aa6ec98f0","Type":"ContainerStarted","Data":"b68680804cdb04592f1345b5a5f14f2cacdb221b9520366bb77025945aa181ee"}
Sep 11 20:01:53.904653 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:53.904593    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:01:53.906090 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:53.906067    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-95wcq" event={"ID":"2936adde-ce1b-4859-858d-db3584a92609","Type":"ContainerStarted","Data":"94ba7ce197c9be4e52132e8c2364ed9dc1d364ac8f8968319417bc76d907f912"}
Sep 11 20:01:53.907836 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:53.907813    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fp6qw" event={"ID":"dda12783-5f3e-4a3e-9fde-4681797a7b60","Type":"ContainerStarted","Data":"9a669341b6c14e828155ad5e959a45dd57b2a4c3f0ef070acb7aabbbebf964b4"}
Sep 11 20:01:53.920550 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:53.920488    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lw7xn" podStartSLOduration=16.471059362 podStartE2EDuration="17.920471041s" totalImagesPullingTime="1.328235044s" totalInitContainerRuntime="121.176635ms" isStatefulPod=true podCreationTimestamp="2026-09-11 20:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:53.918693945 +0000 UTC m=+83.910856215" watchObservedRunningTime="2026-09-11 20:01:53.920471041 +0000 UTC m=+83.912633311"
Sep 11 20:01:53.936074 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:53.936033    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mrmpn" podStartSLOduration=75.054415656 podStartE2EDuration="1m23.936019783s" totalImagesPullingTime="8.881604127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:53.935202196 +0000 UTC m=+83.927364468" watchObservedRunningTime="2026-09-11 20:01:53.936019783 +0000 UTC m=+83.928182053"
Sep 11 20:01:53.955165 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:53.955123    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-4n8lq" podStartSLOduration=8.999714759 podStartE2EDuration="17.955109189s" totalImagesPullingTime="8.95539443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:53.954216643 +0000 UTC m=+83.946378914" watchObservedRunningTime="2026-09-11 20:01:53.955109189 +0000 UTC m=+83.947271459"
Sep 11 20:01:53.972897 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:53.972855    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-95wcq" podStartSLOduration=74.795503188 podStartE2EDuration="1m23.97284108s" totalImagesPullingTime="9.177337892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:53.972261132 +0000 UTC m=+83.964423405" watchObservedRunningTime="2026-09-11 20:01:53.97284108 +0000 UTC m=+83.965003372"
Sep 11 20:01:53.991606 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:53.991564    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fp6qw" podStartSLOduration=75.121836771 podStartE2EDuration="1m23.991549786s" totalImagesPullingTime="8.869713015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:53.990877356 +0000 UTC m=+83.983039627" watchObservedRunningTime="2026-09-11 20:01:53.991549786 +0000 UTC m=+83.983712056"
Sep 11 20:01:54.011346 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:01:54.011301    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-tcl2z" podStartSLOduration=9.177845001 podStartE2EDuration="18.011287349s" totalImagesPullingTime="8.833442348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:54.009873074 +0000 UTC m=+84.002035355" watchObservedRunningTime="2026-09-11 20:01:54.011287349 +0000 UTC m=+84.003449641"
Sep 11 20:02:01.451698 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:01.451667    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:02:01.452156 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:01.451720    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:02:08.849895 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:08.849857    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56dc696459-kgzlz" podUID="8d55865b-db55-4e36-ad84-52b91e019f84" containerName="console" containerID="cri-o://4b363d6634c2b2d9361177f8f5a2108b8d2b42226c8e4a2764b571f5b2f7e1bd" gracePeriod=15
Sep 11 20:02:08.963901 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:08.963876    2580 generic.go:356] "Generic (PLEG): container finished" podID="8d55865b-db55-4e36-ad84-52b91e019f84" containerID="4b363d6634c2b2d9361177f8f5a2108b8d2b42226c8e4a2764b571f5b2f7e1bd" exitCode=0
Sep 11 20:02:08.964036 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:08.963923    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56dc696459-kgzlz" event={"ID":"8d55865b-db55-4e36-ad84-52b91e019f84","Type":"ContainerDied","Data":"4b363d6634c2b2d9361177f8f5a2108b8d2b42226c8e4a2764b571f5b2f7e1bd"}
Sep 11 20:02:09.105632 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.105572    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:02:09.206478 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.206448    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqwmc\" (UniqueName: \"kubernetes.io/projected/8d55865b-db55-4e36-ad84-52b91e019f84-kube-api-access-mqwmc\") pod \"8d55865b-db55-4e36-ad84-52b91e019f84\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") "
Sep 11 20:02:09.206652 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.206488    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d55865b-db55-4e36-ad84-52b91e019f84-console-oauth-config\") pod \"8d55865b-db55-4e36-ad84-52b91e019f84\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") "
Sep 11 20:02:09.206652 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.206543    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d55865b-db55-4e36-ad84-52b91e019f84-oauth-serving-cert\") pod \"8d55865b-db55-4e36-ad84-52b91e019f84\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") "
Sep 11 20:02:09.206652 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.206589    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d55865b-db55-4e36-ad84-52b91e019f84-console-config\") pod \"8d55865b-db55-4e36-ad84-52b91e019f84\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") "
Sep 11 20:02:09.206652 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.206617    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d55865b-db55-4e36-ad84-52b91e019f84-trusted-ca-bundle\") pod \"8d55865b-db55-4e36-ad84-52b91e019f84\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") "
Sep 11 20:02:09.206852 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.206672    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d55865b-db55-4e36-ad84-52b91e019f84-service-ca\") pod \"8d55865b-db55-4e36-ad84-52b91e019f84\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") "
Sep 11 20:02:09.206852 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.206703    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d55865b-db55-4e36-ad84-52b91e019f84-console-serving-cert\") pod \"8d55865b-db55-4e36-ad84-52b91e019f84\" (UID: \"8d55865b-db55-4e36-ad84-52b91e019f84\") "
Sep 11 20:02:09.207005 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.206972    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d55865b-db55-4e36-ad84-52b91e019f84-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8d55865b-db55-4e36-ad84-52b91e019f84" (UID: "8d55865b-db55-4e36-ad84-52b91e019f84"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:02:09.207133 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.207032    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d55865b-db55-4e36-ad84-52b91e019f84-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8d55865b-db55-4e36-ad84-52b91e019f84" (UID: "8d55865b-db55-4e36-ad84-52b91e019f84"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:02:09.207133 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.207031    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d55865b-db55-4e36-ad84-52b91e019f84-console-config" (OuterVolumeSpecName: "console-config") pod "8d55865b-db55-4e36-ad84-52b91e019f84" (UID: "8d55865b-db55-4e36-ad84-52b91e019f84"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:02:09.207133 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.207110    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d55865b-db55-4e36-ad84-52b91e019f84-service-ca" (OuterVolumeSpecName: "service-ca") pod "8d55865b-db55-4e36-ad84-52b91e019f84" (UID: "8d55865b-db55-4e36-ad84-52b91e019f84"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:02:09.208818 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.208788    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d55865b-db55-4e36-ad84-52b91e019f84-kube-api-access-mqwmc" (OuterVolumeSpecName: "kube-api-access-mqwmc") pod "8d55865b-db55-4e36-ad84-52b91e019f84" (UID: "8d55865b-db55-4e36-ad84-52b91e019f84"). InnerVolumeSpecName "kube-api-access-mqwmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:02:09.208818 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.208806    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d55865b-db55-4e36-ad84-52b91e019f84-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8d55865b-db55-4e36-ad84-52b91e019f84" (UID: "8d55865b-db55-4e36-ad84-52b91e019f84"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:02:09.208939 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.208842    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d55865b-db55-4e36-ad84-52b91e019f84-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8d55865b-db55-4e36-ad84-52b91e019f84" (UID: "8d55865b-db55-4e36-ad84-52b91e019f84"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:02:09.308095 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.308068    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqwmc\" (UniqueName: \"kubernetes.io/projected/8d55865b-db55-4e36-ad84-52b91e019f84-kube-api-access-mqwmc\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:02:09.308095 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.308095    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d55865b-db55-4e36-ad84-52b91e019f84-console-oauth-config\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:02:09.308261 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.308104    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d55865b-db55-4e36-ad84-52b91e019f84-oauth-serving-cert\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:02:09.308261 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.308113    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d55865b-db55-4e36-ad84-52b91e019f84-console-config\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:02:09.308261 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.308121    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d55865b-db55-4e36-ad84-52b91e019f84-trusted-ca-bundle\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:02:09.308261 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.308150    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d55865b-db55-4e36-ad84-52b91e019f84-service-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:02:09.308261 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.308160    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d55865b-db55-4e36-ad84-52b91e019f84-console-serving-cert\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:02:09.968432 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.968398    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56dc696459-kgzlz" event={"ID":"8d55865b-db55-4e36-ad84-52b91e019f84","Type":"ContainerDied","Data":"7d86b68a05a51189c23d9d9959af58a51ff4b99bf4c5b5ea7cec164562de2e78"}
Sep 11 20:02:09.968953 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.968463    2580 scope.go:117] "RemoveContainer" containerID="4b363d6634c2b2d9361177f8f5a2108b8d2b42226c8e4a2764b571f5b2f7e1bd"
Sep 11 20:02:09.968953 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.968407    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56dc696459-kgzlz"
Sep 11 20:02:09.988761 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.988737    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56dc696459-kgzlz"]
Sep 11 20:02:09.993042 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:09.993023    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56dc696459-kgzlz"]
Sep 11 20:02:10.438699 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:10.438673    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d55865b-db55-4e36-ad84-52b91e019f84" path="/var/lib/kubelet/pods/8d55865b-db55-4e36-ad84-52b91e019f84/volumes"
Sep 11 20:02:21.456862 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:21.456833    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:02:21.460764 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:21.460743    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-954bf79db-prgff"
Sep 11 20:02:24.914459 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:02:24.914427    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mrmpn"
Sep 11 20:03:04.307877 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.307847    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f774c7bcc-wmdtm"]
Sep 11 20:03:04.308366 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.308115    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d55865b-db55-4e36-ad84-52b91e019f84" containerName="console"
Sep 11 20:03:04.308366 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.308127    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d55865b-db55-4e36-ad84-52b91e019f84" containerName="console"
Sep 11 20:03:04.308366 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.308176    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d55865b-db55-4e36-ad84-52b91e019f84" containerName="console"
Sep 11 20:03:04.310111 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.310083    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:04.312784 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.312760    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 20:03:04.313925 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.313897    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 20:03:04.313925 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.313914    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-g7fth\""
Sep 11 20:03:04.314068 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.314006    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 20:03:04.314068 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.314013    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 20:03:04.314244 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.314228    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 20:03:04.322709 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.322665    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 20:03:04.324104 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.324077    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f774c7bcc-wmdtm"]
Sep 11 20:03:04.393353 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.393333    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3518d484-299b-49a6-882a-abfca88c7ef2-trusted-ca-bundle\") pod \"console-7f774c7bcc-wmdtm\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") " pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:04.393465 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.393361    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3518d484-299b-49a6-882a-abfca88c7ef2-service-ca\") pod \"console-7f774c7bcc-wmdtm\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") " pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:04.393465 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.393380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3518d484-299b-49a6-882a-abfca88c7ef2-console-serving-cert\") pod \"console-7f774c7bcc-wmdtm\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") " pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:04.393465 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.393397    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3518d484-299b-49a6-882a-abfca88c7ef2-console-oauth-config\") pod \"console-7f774c7bcc-wmdtm\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") " pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:04.393591 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.393470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3518d484-299b-49a6-882a-abfca88c7ef2-oauth-serving-cert\") pod \"console-7f774c7bcc-wmdtm\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") " pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:04.393591 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.393516    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwxjf\" (UniqueName: \"kubernetes.io/projected/3518d484-299b-49a6-882a-abfca88c7ef2-kube-api-access-cwxjf\") pod \"console-7f774c7bcc-wmdtm\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") " pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:04.393591 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.393534    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3518d484-299b-49a6-882a-abfca88c7ef2-console-config\") pod \"console-7f774c7bcc-wmdtm\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") " pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:04.494301 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.494274    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3518d484-299b-49a6-882a-abfca88c7ef2-trusted-ca-bundle\") pod \"console-7f774c7bcc-wmdtm\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") " pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:04.494402 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.494304    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3518d484-299b-49a6-882a-abfca88c7ef2-service-ca\") pod \"console-7f774c7bcc-wmdtm\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") " pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:04.494402 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.494322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3518d484-299b-49a6-882a-abfca88c7ef2-console-serving-cert\") pod \"console-7f774c7bcc-wmdtm\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") " pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:04.494402 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.494338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3518d484-299b-49a6-882a-abfca88c7ef2-console-oauth-config\") pod \"console-7f774c7bcc-wmdtm\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") " pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:04.494584 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.494482    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3518d484-299b-49a6-882a-abfca88c7ef2-oauth-serving-cert\") pod \"console-7f774c7bcc-wmdtm\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") " pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:04.494584 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.494562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwxjf\" (UniqueName: \"kubernetes.io/projected/3518d484-299b-49a6-882a-abfca88c7ef2-kube-api-access-cwxjf\") pod \"console-7f774c7bcc-wmdtm\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") " pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:04.494684 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.494594    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3518d484-299b-49a6-882a-abfca88c7ef2-console-config\") pod \"console-7f774c7bcc-wmdtm\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") " pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:04.495087 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.495066    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3518d484-299b-49a6-882a-abfca88c7ef2-service-ca\") pod \"console-7f774c7bcc-wmdtm\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") " pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:04.495244 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.495222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3518d484-299b-49a6-882a-abfca88c7ef2-oauth-serving-cert\") pod \"console-7f774c7bcc-wmdtm\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") " pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:04.495288 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.495264    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3518d484-299b-49a6-882a-abfca88c7ef2-trusted-ca-bundle\") pod \"console-7f774c7bcc-wmdtm\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") " pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:04.495288 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.495275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3518d484-299b-49a6-882a-abfca88c7ef2-console-config\") pod \"console-7f774c7bcc-wmdtm\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") " pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:04.496862 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.496837    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3518d484-299b-49a6-882a-abfca88c7ef2-console-serving-cert\") pod \"console-7f774c7bcc-wmdtm\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") " pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:04.496958 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.496931    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3518d484-299b-49a6-882a-abfca88c7ef2-console-oauth-config\") pod \"console-7f774c7bcc-wmdtm\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") " pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:04.504057 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.504035    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwxjf\" (UniqueName: \"kubernetes.io/projected/3518d484-299b-49a6-882a-abfca88c7ef2-kube-api-access-cwxjf\") pod \"console-7f774c7bcc-wmdtm\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") " pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:04.620762 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.620700    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:04.889336 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:04.889312    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f774c7bcc-wmdtm"]
Sep 11 20:03:04.891698 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:03:04.891668    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3518d484_299b_49a6_882a_abfca88c7ef2.slice/crio-0d50ee1fbc68d674c96d653a31c85f3c98ad50b2b0b31a945e2f71f24acb3225 WatchSource:0}: Error finding container 0d50ee1fbc68d674c96d653a31c85f3c98ad50b2b0b31a945e2f71f24acb3225: Status 404 returned error can't find the container with id 0d50ee1fbc68d674c96d653a31c85f3c98ad50b2b0b31a945e2f71f24acb3225
Sep 11 20:03:05.159262 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:05.159183    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f774c7bcc-wmdtm" event={"ID":"3518d484-299b-49a6-882a-abfca88c7ef2","Type":"ContainerStarted","Data":"079288a4b32ab83e8d0753c6108daa0f2fa9ea85987815aacbd4a6f361be3e7c"}
Sep 11 20:03:05.159262 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:05.159234    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f774c7bcc-wmdtm" event={"ID":"3518d484-299b-49a6-882a-abfca88c7ef2","Type":"ContainerStarted","Data":"0d50ee1fbc68d674c96d653a31c85f3c98ad50b2b0b31a945e2f71f24acb3225"}
Sep 11 20:03:05.177343 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:05.177298    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f774c7bcc-wmdtm" podStartSLOduration=1.177282597 podStartE2EDuration="1.177282597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:03:05.176732701 +0000 UTC m=+155.168894972" watchObservedRunningTime="2026-09-11 20:03:05.177282597 +0000 UTC m=+155.169444868"
Sep 11 20:03:14.621615 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:14.621584    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:14.622073 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:14.621635    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:14.626456 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:14.626436    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:03:15.194685 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:03:15.194649    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:04:15.115598 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.115561    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cf8bb5bb-6ht2v"]
Sep 11 20:04:15.117881 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.117857    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:15.130422 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.130400    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cf8bb5bb-6ht2v"]
Sep 11 20:04:15.279802 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.279769    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-trusted-ca-bundle\") pod \"console-6cf8bb5bb-6ht2v\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") " pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:15.279802 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.279804    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-console-config\") pod \"console-6cf8bb5bb-6ht2v\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") " pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:15.279987 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.279828    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-console-serving-cert\") pod \"console-6cf8bb5bb-6ht2v\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") " pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:15.279987 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.279848    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-console-oauth-config\") pod \"console-6cf8bb5bb-6ht2v\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") " pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:15.279987 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.279866    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-oauth-serving-cert\") pod \"console-6cf8bb5bb-6ht2v\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") " pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:15.279987 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.279888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnmbx\" (UniqueName: \"kubernetes.io/projected/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-kube-api-access-dnmbx\") pod \"console-6cf8bb5bb-6ht2v\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") " pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:15.279987 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.279912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-service-ca\") pod \"console-6cf8bb5bb-6ht2v\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") " pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:15.380624 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.380548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-service-ca\") pod \"console-6cf8bb5bb-6ht2v\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") " pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:15.380624 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.380586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-trusted-ca-bundle\") pod \"console-6cf8bb5bb-6ht2v\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") " pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:15.380624 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.380606    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-console-config\") pod \"console-6cf8bb5bb-6ht2v\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") " pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:15.380891 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.380867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-console-serving-cert\") pod \"console-6cf8bb5bb-6ht2v\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") " pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:15.380956 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.380906    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-console-oauth-config\") pod \"console-6cf8bb5bb-6ht2v\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") " pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:15.380956 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.380924    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-oauth-serving-cert\") pod \"console-6cf8bb5bb-6ht2v\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") " pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:15.380956 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.380948    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnmbx\" (UniqueName: \"kubernetes.io/projected/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-kube-api-access-dnmbx\") pod \"console-6cf8bb5bb-6ht2v\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") " pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:15.381362 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.381342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-service-ca\") pod \"console-6cf8bb5bb-6ht2v\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") " pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:15.381440 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.381409    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-console-config\") pod \"console-6cf8bb5bb-6ht2v\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") " pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:15.381653 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.381635    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-oauth-serving-cert\") pod \"console-6cf8bb5bb-6ht2v\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") " pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:15.381786 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.381766    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-trusted-ca-bundle\") pod \"console-6cf8bb5bb-6ht2v\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") " pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:15.383256 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.383238    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-console-oauth-config\") pod \"console-6cf8bb5bb-6ht2v\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") " pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:15.383365 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.383345    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-console-serving-cert\") pod \"console-6cf8bb5bb-6ht2v\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") " pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:15.388351 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.388328    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnmbx\" (UniqueName: \"kubernetes.io/projected/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-kube-api-access-dnmbx\") pod \"console-6cf8bb5bb-6ht2v\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") " pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:15.426528 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.426506    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:15.693627 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:15.693486    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cf8bb5bb-6ht2v"]
Sep 11 20:04:15.695957 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:04:15.695928    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b54c5d7_07cf_4ee1_9903_77cd6ac42fc4.slice/crio-e24f3a15695ee1fa77d1d8f548784ccccd4173b59bd73c2358bf8386c1548f24 WatchSource:0}: Error finding container e24f3a15695ee1fa77d1d8f548784ccccd4173b59bd73c2358bf8386c1548f24: Status 404 returned error can't find the container with id e24f3a15695ee1fa77d1d8f548784ccccd4173b59bd73c2358bf8386c1548f24
Sep 11 20:04:16.399923 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:16.399884    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf8bb5bb-6ht2v" event={"ID":"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4","Type":"ContainerStarted","Data":"91e15a8b57b39428d9c9981da59b16839921fdb83c4049eae6dc7a2da62fd5b7"}
Sep 11 20:04:16.400378 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:16.399935    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf8bb5bb-6ht2v" event={"ID":"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4","Type":"ContainerStarted","Data":"e24f3a15695ee1fa77d1d8f548784ccccd4173b59bd73c2358bf8386c1548f24"}
Sep 11 20:04:16.416869 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:16.416818    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6cf8bb5bb-6ht2v" podStartSLOduration=1.416801277 podStartE2EDuration="1.416801277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:04:16.415299833 +0000 UTC m=+226.407462116" watchObservedRunningTime="2026-09-11 20:04:16.416801277 +0000 UTC m=+226.408963547"
Sep 11 20:04:25.427443 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:25.427407    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:25.427831 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:25.427459    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:25.432243 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:25.432221    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:25.435846 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:25.435827    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:04:25.492986 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:25.492960    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f774c7bcc-wmdtm"]
Sep 11 20:04:50.512166 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:50.512102    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f774c7bcc-wmdtm" podUID="3518d484-299b-49a6-882a-abfca88c7ef2" containerName="console" containerID="cri-o://079288a4b32ab83e8d0753c6108daa0f2fa9ea85987815aacbd4a6f361be3e7c" gracePeriod=15
Sep 11 20:04:50.753953 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:50.753930    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:04:50.838907 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:50.838824    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3518d484-299b-49a6-882a-abfca88c7ef2-trusted-ca-bundle\") pod \"3518d484-299b-49a6-882a-abfca88c7ef2\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") "
Sep 11 20:04:50.838907 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:50.838890    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3518d484-299b-49a6-882a-abfca88c7ef2-oauth-serving-cert\") pod \"3518d484-299b-49a6-882a-abfca88c7ef2\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") "
Sep 11 20:04:50.838907 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:50.838906    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwxjf\" (UniqueName: \"kubernetes.io/projected/3518d484-299b-49a6-882a-abfca88c7ef2-kube-api-access-cwxjf\") pod \"3518d484-299b-49a6-882a-abfca88c7ef2\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") "
Sep 11 20:04:50.839113 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:50.839008    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3518d484-299b-49a6-882a-abfca88c7ef2-service-ca\") pod \"3518d484-299b-49a6-882a-abfca88c7ef2\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") "
Sep 11 20:04:50.839113 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:50.839044    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3518d484-299b-49a6-882a-abfca88c7ef2-console-serving-cert\") pod \"3518d484-299b-49a6-882a-abfca88c7ef2\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") "
Sep 11 20:04:50.839113 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:50.839088    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3518d484-299b-49a6-882a-abfca88c7ef2-console-oauth-config\") pod \"3518d484-299b-49a6-882a-abfca88c7ef2\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") "
Sep 11 20:04:50.839262 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:50.839125    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3518d484-299b-49a6-882a-abfca88c7ef2-console-config\") pod \"3518d484-299b-49a6-882a-abfca88c7ef2\" (UID: \"3518d484-299b-49a6-882a-abfca88c7ef2\") "
Sep 11 20:04:50.839318 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:50.839256    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3518d484-299b-49a6-882a-abfca88c7ef2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3518d484-299b-49a6-882a-abfca88c7ef2" (UID: "3518d484-299b-49a6-882a-abfca88c7ef2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:04:50.839394 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:50.839369    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3518d484-299b-49a6-882a-abfca88c7ef2-trusted-ca-bundle\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:04:50.839453 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:50.839389    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3518d484-299b-49a6-882a-abfca88c7ef2-service-ca" (OuterVolumeSpecName: "service-ca") pod "3518d484-299b-49a6-882a-abfca88c7ef2" (UID: "3518d484-299b-49a6-882a-abfca88c7ef2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:04:50.839453 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:50.839395    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3518d484-299b-49a6-882a-abfca88c7ef2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3518d484-299b-49a6-882a-abfca88c7ef2" (UID: "3518d484-299b-49a6-882a-abfca88c7ef2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:04:50.839631 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:50.839613    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3518d484-299b-49a6-882a-abfca88c7ef2-console-config" (OuterVolumeSpecName: "console-config") pod "3518d484-299b-49a6-882a-abfca88c7ef2" (UID: "3518d484-299b-49a6-882a-abfca88c7ef2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:04:50.841097 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:50.841071    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3518d484-299b-49a6-882a-abfca88c7ef2-kube-api-access-cwxjf" (OuterVolumeSpecName: "kube-api-access-cwxjf") pod "3518d484-299b-49a6-882a-abfca88c7ef2" (UID: "3518d484-299b-49a6-882a-abfca88c7ef2"). InnerVolumeSpecName "kube-api-access-cwxjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:04:50.841184 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:50.841071    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3518d484-299b-49a6-882a-abfca88c7ef2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3518d484-299b-49a6-882a-abfca88c7ef2" (UID: "3518d484-299b-49a6-882a-abfca88c7ef2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:04:50.841184 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:50.841140    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3518d484-299b-49a6-882a-abfca88c7ef2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3518d484-299b-49a6-882a-abfca88c7ef2" (UID: "3518d484-299b-49a6-882a-abfca88c7ef2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:04:50.939704 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:50.939679    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3518d484-299b-49a6-882a-abfca88c7ef2-oauth-serving-cert\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:04:50.939704 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:50.939704    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwxjf\" (UniqueName: \"kubernetes.io/projected/3518d484-299b-49a6-882a-abfca88c7ef2-kube-api-access-cwxjf\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:04:50.939909 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:50.939714    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3518d484-299b-49a6-882a-abfca88c7ef2-service-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:04:50.939909 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:50.939723    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3518d484-299b-49a6-882a-abfca88c7ef2-console-serving-cert\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:04:50.939909 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:50.939731    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3518d484-299b-49a6-882a-abfca88c7ef2-console-oauth-config\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:04:50.939909 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:50.939740    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3518d484-299b-49a6-882a-abfca88c7ef2-console-config\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:04:51.519433 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:51.519399    2580 generic.go:356] "Generic (PLEG): container finished" podID="3518d484-299b-49a6-882a-abfca88c7ef2" containerID="079288a4b32ab83e8d0753c6108daa0f2fa9ea85987815aacbd4a6f361be3e7c" exitCode=0
Sep 11 20:04:51.519840 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:51.519456    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f774c7bcc-wmdtm"
Sep 11 20:04:51.519840 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:51.519459    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f774c7bcc-wmdtm" event={"ID":"3518d484-299b-49a6-882a-abfca88c7ef2","Type":"ContainerDied","Data":"079288a4b32ab83e8d0753c6108daa0f2fa9ea85987815aacbd4a6f361be3e7c"}
Sep 11 20:04:51.519840 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:51.519546    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f774c7bcc-wmdtm" event={"ID":"3518d484-299b-49a6-882a-abfca88c7ef2","Type":"ContainerDied","Data":"0d50ee1fbc68d674c96d653a31c85f3c98ad50b2b0b31a945e2f71f24acb3225"}
Sep 11 20:04:51.519840 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:51.519567    2580 scope.go:117] "RemoveContainer" containerID="079288a4b32ab83e8d0753c6108daa0f2fa9ea85987815aacbd4a6f361be3e7c"
Sep 11 20:04:51.527976 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:51.527955    2580 scope.go:117] "RemoveContainer" containerID="079288a4b32ab83e8d0753c6108daa0f2fa9ea85987815aacbd4a6f361be3e7c"
Sep 11 20:04:51.528305 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:04:51.528283    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"079288a4b32ab83e8d0753c6108daa0f2fa9ea85987815aacbd4a6f361be3e7c\": container with ID starting with 079288a4b32ab83e8d0753c6108daa0f2fa9ea85987815aacbd4a6f361be3e7c not found: ID does not exist" containerID="079288a4b32ab83e8d0753c6108daa0f2fa9ea85987815aacbd4a6f361be3e7c"
Sep 11 20:04:51.528388 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:51.528319    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"079288a4b32ab83e8d0753c6108daa0f2fa9ea85987815aacbd4a6f361be3e7c"} err="failed to get container status \"079288a4b32ab83e8d0753c6108daa0f2fa9ea85987815aacbd4a6f361be3e7c\": rpc error: code = NotFound desc = could not find container \"079288a4b32ab83e8d0753c6108daa0f2fa9ea85987815aacbd4a6f361be3e7c\": container with ID starting with 079288a4b32ab83e8d0753c6108daa0f2fa9ea85987815aacbd4a6f361be3e7c not found: ID does not exist"
Sep 11 20:04:51.556826 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:51.556796    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f774c7bcc-wmdtm"]
Sep 11 20:04:51.561290 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:51.561266    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f774c7bcc-wmdtm"]
Sep 11 20:04:52.438951 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:52.438920    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3518d484-299b-49a6-882a-abfca88c7ef2" path="/var/lib/kubelet/pods/3518d484-299b-49a6-882a-abfca88c7ef2/volumes"
Sep 11 20:04:53.512823 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:53.512791    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5"]
Sep 11 20:04:53.513174 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:53.513040    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3518d484-299b-49a6-882a-abfca88c7ef2" containerName="console"
Sep 11 20:04:53.513174 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:53.513051    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3518d484-299b-49a6-882a-abfca88c7ef2" containerName="console"
Sep 11 20:04:53.513174 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:53.513100    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3518d484-299b-49a6-882a-abfca88c7ef2" containerName="console"
Sep 11 20:04:53.515025 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:53.515009    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5"
Sep 11 20:04:53.520654 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:53.520630    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 20:04:53.520805 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:53.520765    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-g5whd\""
Sep 11 20:04:53.521684 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:53.521665    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 20:04:53.529060 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:53.529039    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5"]
Sep 11 20:04:53.557520 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:53.557477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/33c5ce1e-ad06-42fd-a41a-09832f1f2d2b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5\" (UID: \"33c5ce1e-ad06-42fd-a41a-09832f1f2d2b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5"
Sep 11 20:04:53.557634 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:53.557524    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj8z8\" (UniqueName: \"kubernetes.io/projected/33c5ce1e-ad06-42fd-a41a-09832f1f2d2b-kube-api-access-cj8z8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5\" (UID: \"33c5ce1e-ad06-42fd-a41a-09832f1f2d2b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5"
Sep 11 20:04:53.557634 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:53.557618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/33c5ce1e-ad06-42fd-a41a-09832f1f2d2b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5\" (UID: \"33c5ce1e-ad06-42fd-a41a-09832f1f2d2b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5"
Sep 11 20:04:53.658918 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:53.658882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/33c5ce1e-ad06-42fd-a41a-09832f1f2d2b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5\" (UID: \"33c5ce1e-ad06-42fd-a41a-09832f1f2d2b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5"
Sep 11 20:04:53.658918 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:53.658924    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj8z8\" (UniqueName: \"kubernetes.io/projected/33c5ce1e-ad06-42fd-a41a-09832f1f2d2b-kube-api-access-cj8z8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5\" (UID: \"33c5ce1e-ad06-42fd-a41a-09832f1f2d2b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5"
Sep 11 20:04:53.659123 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:53.659051    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/33c5ce1e-ad06-42fd-a41a-09832f1f2d2b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5\" (UID: \"33c5ce1e-ad06-42fd-a41a-09832f1f2d2b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5"
Sep 11 20:04:53.659328 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:53.659310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/33c5ce1e-ad06-42fd-a41a-09832f1f2d2b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5\" (UID: \"33c5ce1e-ad06-42fd-a41a-09832f1f2d2b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5"
Sep 11 20:04:53.659367 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:53.659338    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/33c5ce1e-ad06-42fd-a41a-09832f1f2d2b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5\" (UID: \"33c5ce1e-ad06-42fd-a41a-09832f1f2d2b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5"
Sep 11 20:04:53.667716 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:53.667690    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj8z8\" (UniqueName: \"kubernetes.io/projected/33c5ce1e-ad06-42fd-a41a-09832f1f2d2b-kube-api-access-cj8z8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5\" (UID: \"33c5ce1e-ad06-42fd-a41a-09832f1f2d2b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5"
Sep 11 20:04:53.823141 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:53.823079    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5"
Sep 11 20:04:54.092620 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:54.092527    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5"]
Sep 11 20:04:54.096557 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:04:54.096525    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c5ce1e_ad06_42fd_a41a_09832f1f2d2b.slice/crio-bc853310306501d964b7f969dee767584424296d150a7c86108888232d17759b WatchSource:0}: Error finding container bc853310306501d964b7f969dee767584424296d150a7c86108888232d17759b: Status 404 returned error can't find the container with id bc853310306501d964b7f969dee767584424296d150a7c86108888232d17759b
Sep 11 20:04:54.532017 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:04:54.531976    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5" event={"ID":"33c5ce1e-ad06-42fd-a41a-09832f1f2d2b","Type":"ContainerStarted","Data":"bc853310306501d964b7f969dee767584424296d150a7c86108888232d17759b"}
Sep 11 20:05:01.559407 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:01.559372    2580 generic.go:356] "Generic (PLEG): container finished" podID="33c5ce1e-ad06-42fd-a41a-09832f1f2d2b" containerID="2cc0b8457670a4651c7f6d379afa07d1ec7d2afc2e4088afd102cf8c80cd5998" exitCode=0
Sep 11 20:05:01.559810 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:01.559429    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5" event={"ID":"33c5ce1e-ad06-42fd-a41a-09832f1f2d2b","Type":"ContainerDied","Data":"2cc0b8457670a4651c7f6d379afa07d1ec7d2afc2e4088afd102cf8c80cd5998"}
Sep 11 20:05:04.572216 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:04.572185    2580 generic.go:356] "Generic (PLEG): container finished" podID="33c5ce1e-ad06-42fd-a41a-09832f1f2d2b" containerID="72329b77716c3a9177f56db1fef011e9491f3669692732eae345b18c79cc79de" exitCode=0
Sep 11 20:05:04.572653 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:04.572253    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5" event={"ID":"33c5ce1e-ad06-42fd-a41a-09832f1f2d2b","Type":"ContainerDied","Data":"72329b77716c3a9177f56db1fef011e9491f3669692732eae345b18c79cc79de"}
Sep 11 20:05:12.603897 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:12.603863    2580 generic.go:356] "Generic (PLEG): container finished" podID="33c5ce1e-ad06-42fd-a41a-09832f1f2d2b" containerID="657396950a0fd255321c32fbaf6e45808cabd82acc2f0cbe75e761838702bcaf" exitCode=0
Sep 11 20:05:12.604270 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:12.603918    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5" event={"ID":"33c5ce1e-ad06-42fd-a41a-09832f1f2d2b","Type":"ContainerDied","Data":"657396950a0fd255321c32fbaf6e45808cabd82acc2f0cbe75e761838702bcaf"}
Sep 11 20:05:13.752349 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:13.752325    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5"
Sep 11 20:05:13.818136 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:13.818099    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cj8z8\" (UniqueName: \"kubernetes.io/projected/33c5ce1e-ad06-42fd-a41a-09832f1f2d2b-kube-api-access-cj8z8\") pod \"33c5ce1e-ad06-42fd-a41a-09832f1f2d2b\" (UID: \"33c5ce1e-ad06-42fd-a41a-09832f1f2d2b\") "
Sep 11 20:05:13.818136 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:13.818135    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/33c5ce1e-ad06-42fd-a41a-09832f1f2d2b-bundle\") pod \"33c5ce1e-ad06-42fd-a41a-09832f1f2d2b\" (UID: \"33c5ce1e-ad06-42fd-a41a-09832f1f2d2b\") "
Sep 11 20:05:13.818345 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:13.818164    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/33c5ce1e-ad06-42fd-a41a-09832f1f2d2b-util\") pod \"33c5ce1e-ad06-42fd-a41a-09832f1f2d2b\" (UID: \"33c5ce1e-ad06-42fd-a41a-09832f1f2d2b\") "
Sep 11 20:05:13.818806 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:13.818783    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c5ce1e-ad06-42fd-a41a-09832f1f2d2b-bundle" (OuterVolumeSpecName: "bundle") pod "33c5ce1e-ad06-42fd-a41a-09832f1f2d2b" (UID: "33c5ce1e-ad06-42fd-a41a-09832f1f2d2b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:05:13.820236 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:13.820210    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c5ce1e-ad06-42fd-a41a-09832f1f2d2b-kube-api-access-cj8z8" (OuterVolumeSpecName: "kube-api-access-cj8z8") pod "33c5ce1e-ad06-42fd-a41a-09832f1f2d2b" (UID: "33c5ce1e-ad06-42fd-a41a-09832f1f2d2b"). InnerVolumeSpecName "kube-api-access-cj8z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:05:13.823039 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:13.823015    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c5ce1e-ad06-42fd-a41a-09832f1f2d2b-util" (OuterVolumeSpecName: "util") pod "33c5ce1e-ad06-42fd-a41a-09832f1f2d2b" (UID: "33c5ce1e-ad06-42fd-a41a-09832f1f2d2b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:05:13.918736 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:13.918663    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj8z8\" (UniqueName: \"kubernetes.io/projected/33c5ce1e-ad06-42fd-a41a-09832f1f2d2b-kube-api-access-cj8z8\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:05:13.918736 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:13.918691    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/33c5ce1e-ad06-42fd-a41a-09832f1f2d2b-bundle\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:05:13.918736 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:13.918700    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/33c5ce1e-ad06-42fd-a41a-09832f1f2d2b-util\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:05:14.612020 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:14.611993    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5"
Sep 11 20:05:14.612020 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:14.612001    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zl9v5" event={"ID":"33c5ce1e-ad06-42fd-a41a-09832f1f2d2b","Type":"ContainerDied","Data":"bc853310306501d964b7f969dee767584424296d150a7c86108888232d17759b"}
Sep 11 20:05:14.612211 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:14.612068    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc853310306501d964b7f969dee767584424296d150a7c86108888232d17759b"
Sep 11 20:05:21.523804 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:21.523767    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7n7dj"]
Sep 11 20:05:21.524300 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:21.524042    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33c5ce1e-ad06-42fd-a41a-09832f1f2d2b" containerName="util"
Sep 11 20:05:21.524300 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:21.524053    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="33c5ce1e-ad06-42fd-a41a-09832f1f2d2b" containerName="util"
Sep 11 20:05:21.524300 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:21.524060    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33c5ce1e-ad06-42fd-a41a-09832f1f2d2b" containerName="pull"
Sep 11 20:05:21.524300 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:21.524066    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="33c5ce1e-ad06-42fd-a41a-09832f1f2d2b" containerName="pull"
Sep 11 20:05:21.524300 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:21.524072    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33c5ce1e-ad06-42fd-a41a-09832f1f2d2b" containerName="extract"
Sep 11 20:05:21.524300 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:21.524078    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="33c5ce1e-ad06-42fd-a41a-09832f1f2d2b" containerName="extract"
Sep 11 20:05:21.524300 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:21.524143    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="33c5ce1e-ad06-42fd-a41a-09832f1f2d2b" containerName="extract"
Sep 11 20:05:21.526024 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:21.526008    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7n7dj"
Sep 11 20:05:21.529047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:21.529021    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 11 20:05:21.529047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:21.529039    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-64qjc\""
Sep 11 20:05:21.529212 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:21.529171    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 11 20:05:21.542147 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:21.542126    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7n7dj"]
Sep 11 20:05:21.668888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:21.668858    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe2c1293-1ad4-4d68-9175-ae3540c9e15e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-7n7dj\" (UID: \"fe2c1293-1ad4-4d68-9175-ae3540c9e15e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7n7dj"
Sep 11 20:05:21.669067 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:21.668902    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vs8d\" (UniqueName: \"kubernetes.io/projected/fe2c1293-1ad4-4d68-9175-ae3540c9e15e-kube-api-access-2vs8d\") pod \"cert-manager-operator-controller-manager-55c444bf86-7n7dj\" (UID: \"fe2c1293-1ad4-4d68-9175-ae3540c9e15e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7n7dj"
Sep 11 20:05:21.770117 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:21.770086    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe2c1293-1ad4-4d68-9175-ae3540c9e15e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-7n7dj\" (UID: \"fe2c1293-1ad4-4d68-9175-ae3540c9e15e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7n7dj"
Sep 11 20:05:21.770254 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:21.770131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vs8d\" (UniqueName: \"kubernetes.io/projected/fe2c1293-1ad4-4d68-9175-ae3540c9e15e-kube-api-access-2vs8d\") pod \"cert-manager-operator-controller-manager-55c444bf86-7n7dj\" (UID: \"fe2c1293-1ad4-4d68-9175-ae3540c9e15e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7n7dj"
Sep 11 20:05:21.770527 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:21.770480    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe2c1293-1ad4-4d68-9175-ae3540c9e15e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-7n7dj\" (UID: \"fe2c1293-1ad4-4d68-9175-ae3540c9e15e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7n7dj"
Sep 11 20:05:21.778813 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:21.778765    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vs8d\" (UniqueName: \"kubernetes.io/projected/fe2c1293-1ad4-4d68-9175-ae3540c9e15e-kube-api-access-2vs8d\") pod \"cert-manager-operator-controller-manager-55c444bf86-7n7dj\" (UID: \"fe2c1293-1ad4-4d68-9175-ae3540c9e15e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7n7dj"
Sep 11 20:05:21.834845 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:21.834822    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7n7dj"
Sep 11 20:05:22.107786 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:22.107759    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7n7dj"]
Sep 11 20:05:22.110036 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:05:22.110008    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe2c1293_1ad4_4d68_9175_ae3540c9e15e.slice/crio-761be82f059efe7912e5bab93c774d6b6e3e770546ea0970a3f94bb629ef0419 WatchSource:0}: Error finding container 761be82f059efe7912e5bab93c774d6b6e3e770546ea0970a3f94bb629ef0419: Status 404 returned error can't find the container with id 761be82f059efe7912e5bab93c774d6b6e3e770546ea0970a3f94bb629ef0419
Sep 11 20:05:22.642023 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:22.641989    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7n7dj" event={"ID":"fe2c1293-1ad4-4d68-9175-ae3540c9e15e","Type":"ContainerStarted","Data":"761be82f059efe7912e5bab93c774d6b6e3e770546ea0970a3f94bb629ef0419"}
Sep 11 20:05:24.651258 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:24.651219    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7n7dj" event={"ID":"fe2c1293-1ad4-4d68-9175-ae3540c9e15e","Type":"ContainerStarted","Data":"60711974af2c190f4c5c6b7bee7ac6433e4e140df654a83745c63dcd6d61c392"}
Sep 11 20:05:24.733843 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:24.733791    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7n7dj" podStartSLOduration=2.080680353 podStartE2EDuration="3.733773965s" totalImagesPullingTime="1.653093612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:05:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:05:24.678928204 +0000 UTC m=+294.671090476" watchObservedRunningTime="2026-09-11 20:05:24.733773965 +0000 UTC m=+294.725936247"
Sep 11 20:05:24.734283 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:24.734268    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jb2cs"]
Sep 11 20:05:24.736663 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:24.736640    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-jb2cs"
Sep 11 20:05:24.739703 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:24.739681    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 20:05:24.739813 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:24.739752    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-npsrr\""
Sep 11 20:05:24.740571 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:24.740556    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 20:05:24.751758 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:24.751735    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jb2cs"]
Sep 11 20:05:24.894126 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:24.894100    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19d35618-657c-42e0-b8ed-e98742eb577a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jb2cs\" (UID: \"19d35618-657c-42e0-b8ed-e98742eb577a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jb2cs"
Sep 11 20:05:24.894255 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:24.894165    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk5sh\" (UniqueName: \"kubernetes.io/projected/19d35618-657c-42e0-b8ed-e98742eb577a-kube-api-access-zk5sh\") pod \"cert-manager-webhook-6486ffd997-jb2cs\" (UID: \"19d35618-657c-42e0-b8ed-e98742eb577a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jb2cs"
Sep 11 20:05:24.995094 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:24.995062    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19d35618-657c-42e0-b8ed-e98742eb577a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jb2cs\" (UID: \"19d35618-657c-42e0-b8ed-e98742eb577a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jb2cs"
Sep 11 20:05:24.995238 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:24.995125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk5sh\" (UniqueName: \"kubernetes.io/projected/19d35618-657c-42e0-b8ed-e98742eb577a-kube-api-access-zk5sh\") pod \"cert-manager-webhook-6486ffd997-jb2cs\" (UID: \"19d35618-657c-42e0-b8ed-e98742eb577a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jb2cs"
Sep 11 20:05:25.009100 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:25.009062    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19d35618-657c-42e0-b8ed-e98742eb577a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jb2cs\" (UID: \"19d35618-657c-42e0-b8ed-e98742eb577a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jb2cs"
Sep 11 20:05:25.011216 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:25.011194    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk5sh\" (UniqueName: \"kubernetes.io/projected/19d35618-657c-42e0-b8ed-e98742eb577a-kube-api-access-zk5sh\") pod \"cert-manager-webhook-6486ffd997-jb2cs\" (UID: \"19d35618-657c-42e0-b8ed-e98742eb577a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jb2cs"
Sep 11 20:05:25.057141 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:25.057118    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-jb2cs"
Sep 11 20:05:25.338834 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:25.338759    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jb2cs"]
Sep 11 20:05:25.341444 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:05:25.341416    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19d35618_657c_42e0_b8ed_e98742eb577a.slice/crio-98afb883a2196e528900f475164e9e395e618ccbfab01b5843d640c70a1f82ea WatchSource:0}: Error finding container 98afb883a2196e528900f475164e9e395e618ccbfab01b5843d640c70a1f82ea: Status 404 returned error can't find the container with id 98afb883a2196e528900f475164e9e395e618ccbfab01b5843d640c70a1f82ea
Sep 11 20:05:25.656307 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:25.656226    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-jb2cs" event={"ID":"19d35618-657c-42e0-b8ed-e98742eb577a","Type":"ContainerStarted","Data":"98afb883a2196e528900f475164e9e395e618ccbfab01b5843d640c70a1f82ea"}
Sep 11 20:05:25.837479 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:25.837445    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf"]
Sep 11 20:05:25.840192 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:25.840170    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf"
Sep 11 20:05:25.843946 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:25.843923    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 20:05:25.844052 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:25.844029    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-g5whd\""
Sep 11 20:05:25.844959 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:25.844942    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 20:05:25.851516 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:25.851482    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf"]
Sep 11 20:05:26.004140 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:26.004107    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v8j9\" (UniqueName: \"kubernetes.io/projected/7700299d-ac1d-48c7-8469-beef0a729c80-kube-api-access-5v8j9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf\" (UID: \"7700299d-ac1d-48c7-8469-beef0a729c80\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf"
Sep 11 20:05:26.004305 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:26.004164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7700299d-ac1d-48c7-8469-beef0a729c80-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf\" (UID: \"7700299d-ac1d-48c7-8469-beef0a729c80\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf"
Sep 11 20:05:26.004305 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:26.004211    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7700299d-ac1d-48c7-8469-beef0a729c80-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf\" (UID: \"7700299d-ac1d-48c7-8469-beef0a729c80\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf"
Sep 11 20:05:26.104944 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:26.104912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7700299d-ac1d-48c7-8469-beef0a729c80-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf\" (UID: \"7700299d-ac1d-48c7-8469-beef0a729c80\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf"
Sep 11 20:05:26.105112 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:26.104957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v8j9\" (UniqueName: \"kubernetes.io/projected/7700299d-ac1d-48c7-8469-beef0a729c80-kube-api-access-5v8j9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf\" (UID: \"7700299d-ac1d-48c7-8469-beef0a729c80\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf"
Sep 11 20:05:26.105112 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:26.105017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7700299d-ac1d-48c7-8469-beef0a729c80-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf\" (UID: \"7700299d-ac1d-48c7-8469-beef0a729c80\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf"
Sep 11 20:05:26.105343 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:26.105322    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7700299d-ac1d-48c7-8469-beef0a729c80-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf\" (UID: \"7700299d-ac1d-48c7-8469-beef0a729c80\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf"
Sep 11 20:05:26.105409 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:26.105354    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7700299d-ac1d-48c7-8469-beef0a729c80-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf\" (UID: \"7700299d-ac1d-48c7-8469-beef0a729c80\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf"
Sep 11 20:05:26.113648 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:26.113627    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v8j9\" (UniqueName: \"kubernetes.io/projected/7700299d-ac1d-48c7-8469-beef0a729c80-kube-api-access-5v8j9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf\" (UID: \"7700299d-ac1d-48c7-8469-beef0a729c80\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf"
Sep 11 20:05:26.149546 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:26.149522    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf"
Sep 11 20:05:26.416743 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:26.416674    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf"]
Sep 11 20:05:26.419980 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:05:26.419952    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7700299d_ac1d_48c7_8469_beef0a729c80.slice/crio-ccfa6e03b1565bad5551127b7b5f6f0993ecef7bf46a6af8610bdf1c95dfa77a WatchSource:0}: Error finding container ccfa6e03b1565bad5551127b7b5f6f0993ecef7bf46a6af8610bdf1c95dfa77a: Status 404 returned error can't find the container with id ccfa6e03b1565bad5551127b7b5f6f0993ecef7bf46a6af8610bdf1c95dfa77a
Sep 11 20:05:26.662967 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:26.662932    2580 generic.go:356] "Generic (PLEG): container finished" podID="7700299d-ac1d-48c7-8469-beef0a729c80" containerID="15e671c448911923dc953570ba465bdef1028354cc05e8a336b90b16aa9b3043" exitCode=0
Sep 11 20:05:26.663410 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:26.662965    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf" event={"ID":"7700299d-ac1d-48c7-8469-beef0a729c80","Type":"ContainerDied","Data":"15e671c448911923dc953570ba465bdef1028354cc05e8a336b90b16aa9b3043"}
Sep 11 20:05:26.663410 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:26.663022    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf" event={"ID":"7700299d-ac1d-48c7-8469-beef0a729c80","Type":"ContainerStarted","Data":"ccfa6e03b1565bad5551127b7b5f6f0993ecef7bf46a6af8610bdf1c95dfa77a"}
Sep 11 20:05:27.331018 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:27.330985    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cgh9z"]
Sep 11 20:05:27.333424 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:27.333402    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cgh9z"
Sep 11 20:05:27.335967 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:27.335948    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-d9npz\""
Sep 11 20:05:27.346013 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:27.345994    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cgh9z"]
Sep 11 20:05:27.417432 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:27.417336    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h4q8\" (UniqueName: \"kubernetes.io/projected/70f01fa4-11ce-4424-8291-6345fba55d8a-kube-api-access-9h4q8\") pod \"cert-manager-cainjector-559cdd8c87-cgh9z\" (UID: \"70f01fa4-11ce-4424-8291-6345fba55d8a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cgh9z"
Sep 11 20:05:27.417432 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:27.417420    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70f01fa4-11ce-4424-8291-6345fba55d8a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cgh9z\" (UID: \"70f01fa4-11ce-4424-8291-6345fba55d8a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cgh9z"
Sep 11 20:05:27.518767 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:27.518733    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h4q8\" (UniqueName: \"kubernetes.io/projected/70f01fa4-11ce-4424-8291-6345fba55d8a-kube-api-access-9h4q8\") pod \"cert-manager-cainjector-559cdd8c87-cgh9z\" (UID: \"70f01fa4-11ce-4424-8291-6345fba55d8a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cgh9z"
Sep 11 20:05:27.518924 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:27.518835    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70f01fa4-11ce-4424-8291-6345fba55d8a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cgh9z\" (UID: \"70f01fa4-11ce-4424-8291-6345fba55d8a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cgh9z"
Sep 11 20:05:27.526585 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:27.526559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70f01fa4-11ce-4424-8291-6345fba55d8a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cgh9z\" (UID: \"70f01fa4-11ce-4424-8291-6345fba55d8a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cgh9z"
Sep 11 20:05:27.526834 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:27.526813    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h4q8\" (UniqueName: \"kubernetes.io/projected/70f01fa4-11ce-4424-8291-6345fba55d8a-kube-api-access-9h4q8\") pod \"cert-manager-cainjector-559cdd8c87-cgh9z\" (UID: \"70f01fa4-11ce-4424-8291-6345fba55d8a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cgh9z"
Sep 11 20:05:27.643632 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:27.643596    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cgh9z"
Sep 11 20:05:28.612691 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:28.612504    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cgh9z"]
Sep 11 20:05:28.614874 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:05:28.614849    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70f01fa4_11ce_4424_8291_6345fba55d8a.slice/crio-9b8903557fddbf27f22b5413aaffd31751b2d59562f86f13fb0d27c76cb07737 WatchSource:0}: Error finding container 9b8903557fddbf27f22b5413aaffd31751b2d59562f86f13fb0d27c76cb07737: Status 404 returned error can't find the container with id 9b8903557fddbf27f22b5413aaffd31751b2d59562f86f13fb0d27c76cb07737
Sep 11 20:05:28.673605 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:28.673529    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-jb2cs" event={"ID":"19d35618-657c-42e0-b8ed-e98742eb577a","Type":"ContainerStarted","Data":"b72ebedc88dbec025cf23efce8a2728dd2cf47f73e63261d374e2a80cafe0382"}
Sep 11 20:05:28.673605 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:28.673589    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-jb2cs"
Sep 11 20:05:28.675421 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:28.675398    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cgh9z" event={"ID":"70f01fa4-11ce-4424-8291-6345fba55d8a","Type":"ContainerStarted","Data":"9b8903557fddbf27f22b5413aaffd31751b2d59562f86f13fb0d27c76cb07737"}
Sep 11 20:05:28.690375 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:28.690330    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-jb2cs" podStartSLOduration=1.652622444 podStartE2EDuration="4.690311861s" totalImagesPullingTime="3.037689417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:05:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:05:28.68848645 +0000 UTC m=+298.680648721" watchObservedRunningTime="2026-09-11 20:05:28.690311861 +0000 UTC m=+298.682474131"
Sep 11 20:05:29.683040 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:29.683005    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cgh9z" event={"ID":"70f01fa4-11ce-4424-8291-6345fba55d8a","Type":"ContainerStarted","Data":"55e222db8f1cb934d6e7cfa87fc6906b0c2dee72c55f6102a7b90ecda9dd3477"}
Sep 11 20:05:29.698949 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:29.698901    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cgh9z" podStartSLOduration=2.698884751 podStartE2EDuration="2.698884751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:05:29.698379727 +0000 UTC m=+299.690541998" watchObservedRunningTime="2026-09-11 20:05:29.698884751 +0000 UTC m=+299.691047020"
Sep 11 20:05:30.363643 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:30.363615    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 20:05:30.689272 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:30.689237    2580 generic.go:356] "Generic (PLEG): container finished" podID="7700299d-ac1d-48c7-8469-beef0a729c80" containerID="aa39f8a658dcbb35f94618f53c5666ff38cb5a90f656aa33b9c288eabea26f55" exitCode=0
Sep 11 20:05:30.691520 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:30.689323    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf" event={"ID":"7700299d-ac1d-48c7-8469-beef0a729c80","Type":"ContainerDied","Data":"aa39f8a658dcbb35f94618f53c5666ff38cb5a90f656aa33b9c288eabea26f55"}
Sep 11 20:05:31.695822 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:31.695791    2580 generic.go:356] "Generic (PLEG): container finished" podID="7700299d-ac1d-48c7-8469-beef0a729c80" containerID="b45f9c875a0426a76e34720c12b87aaf55829f0a21ea311384a1cfde9b4c8a45" exitCode=0
Sep 11 20:05:31.696213 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:31.695850    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf" event={"ID":"7700299d-ac1d-48c7-8469-beef0a729c80","Type":"ContainerDied","Data":"b45f9c875a0426a76e34720c12b87aaf55829f0a21ea311384a1cfde9b4c8a45"}
Sep 11 20:05:32.820860 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:32.820836    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf"
Sep 11 20:05:32.963150 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:32.963049    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5v8j9\" (UniqueName: \"kubernetes.io/projected/7700299d-ac1d-48c7-8469-beef0a729c80-kube-api-access-5v8j9\") pod \"7700299d-ac1d-48c7-8469-beef0a729c80\" (UID: \"7700299d-ac1d-48c7-8469-beef0a729c80\") "
Sep 11 20:05:32.963150 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:32.963112    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7700299d-ac1d-48c7-8469-beef0a729c80-util\") pod \"7700299d-ac1d-48c7-8469-beef0a729c80\" (UID: \"7700299d-ac1d-48c7-8469-beef0a729c80\") "
Sep 11 20:05:32.963150 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:32.963149    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7700299d-ac1d-48c7-8469-beef0a729c80-bundle\") pod \"7700299d-ac1d-48c7-8469-beef0a729c80\" (UID: \"7700299d-ac1d-48c7-8469-beef0a729c80\") "
Sep 11 20:05:32.963572 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:32.963547    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7700299d-ac1d-48c7-8469-beef0a729c80-bundle" (OuterVolumeSpecName: "bundle") pod "7700299d-ac1d-48c7-8469-beef0a729c80" (UID: "7700299d-ac1d-48c7-8469-beef0a729c80"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:05:32.965286 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:32.965260    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7700299d-ac1d-48c7-8469-beef0a729c80-kube-api-access-5v8j9" (OuterVolumeSpecName: "kube-api-access-5v8j9") pod "7700299d-ac1d-48c7-8469-beef0a729c80" (UID: "7700299d-ac1d-48c7-8469-beef0a729c80"). InnerVolumeSpecName "kube-api-access-5v8j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:05:32.967737 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:32.967714    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7700299d-ac1d-48c7-8469-beef0a729c80-util" (OuterVolumeSpecName: "util") pod "7700299d-ac1d-48c7-8469-beef0a729c80" (UID: "7700299d-ac1d-48c7-8469-beef0a729c80"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:05:33.064387 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:33.064349    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v8j9\" (UniqueName: \"kubernetes.io/projected/7700299d-ac1d-48c7-8469-beef0a729c80-kube-api-access-5v8j9\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:05:33.064387 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:33.064382    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7700299d-ac1d-48c7-8469-beef0a729c80-util\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:05:33.064387 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:33.064392    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7700299d-ac1d-48c7-8469-beef0a729c80-bundle\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:05:33.705585 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:33.705546    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf"
Sep 11 20:05:33.705754 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:33.705549    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff6vxf" event={"ID":"7700299d-ac1d-48c7-8469-beef0a729c80","Type":"ContainerDied","Data":"ccfa6e03b1565bad5551127b7b5f6f0993ecef7bf46a6af8610bdf1c95dfa77a"}
Sep 11 20:05:33.705754 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:33.705671    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccfa6e03b1565bad5551127b7b5f6f0993ecef7bf46a6af8610bdf1c95dfa77a"
Sep 11 20:05:34.685747 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:34.685715    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-jb2cs"
Sep 11 20:05:48.990342 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:48.990301    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42"]
Sep 11 20:05:48.990806 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:48.990574    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7700299d-ac1d-48c7-8469-beef0a729c80" containerName="extract"
Sep 11 20:05:48.990806 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:48.990585    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7700299d-ac1d-48c7-8469-beef0a729c80" containerName="extract"
Sep 11 20:05:48.990806 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:48.990605    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7700299d-ac1d-48c7-8469-beef0a729c80" containerName="pull"
Sep 11 20:05:48.990806 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:48.990610    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7700299d-ac1d-48c7-8469-beef0a729c80" containerName="pull"
Sep 11 20:05:48.990806 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:48.990617    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7700299d-ac1d-48c7-8469-beef0a729c80" containerName="util"
Sep 11 20:05:48.990806 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:48.990623    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7700299d-ac1d-48c7-8469-beef0a729c80" containerName="util"
Sep 11 20:05:48.990806 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:48.990689    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7700299d-ac1d-48c7-8469-beef0a729c80" containerName="extract"
Sep 11 20:05:48.994855 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:48.994834    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42"
Sep 11 20:05:48.997649 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:48.997628    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 20:05:48.997742 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:48.997679    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 20:05:48.998819 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:48.998799    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-g5whd\""
Sep 11 20:05:49.003740 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:49.003719    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42"]
Sep 11 20:05:49.084391 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:49.084354    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb6c7\" (UniqueName: \"kubernetes.io/projected/aeab8027-b217-476e-aa9f-bfc80ce02b93-kube-api-access-vb6c7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42\" (UID: \"aeab8027-b217-476e-aa9f-bfc80ce02b93\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42"
Sep 11 20:05:49.084391 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:49.084402    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aeab8027-b217-476e-aa9f-bfc80ce02b93-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42\" (UID: \"aeab8027-b217-476e-aa9f-bfc80ce02b93\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42"
Sep 11 20:05:49.084649 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:49.084437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aeab8027-b217-476e-aa9f-bfc80ce02b93-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42\" (UID: \"aeab8027-b217-476e-aa9f-bfc80ce02b93\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42"
Sep 11 20:05:49.185626 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:49.185598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb6c7\" (UniqueName: \"kubernetes.io/projected/aeab8027-b217-476e-aa9f-bfc80ce02b93-kube-api-access-vb6c7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42\" (UID: \"aeab8027-b217-476e-aa9f-bfc80ce02b93\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42"
Sep 11 20:05:49.185762 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:49.185639    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aeab8027-b217-476e-aa9f-bfc80ce02b93-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42\" (UID: \"aeab8027-b217-476e-aa9f-bfc80ce02b93\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42"
Sep 11 20:05:49.185762 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:49.185670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aeab8027-b217-476e-aa9f-bfc80ce02b93-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42\" (UID: \"aeab8027-b217-476e-aa9f-bfc80ce02b93\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42"
Sep 11 20:05:49.185982 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:49.185964    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aeab8027-b217-476e-aa9f-bfc80ce02b93-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42\" (UID: \"aeab8027-b217-476e-aa9f-bfc80ce02b93\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42"
Sep 11 20:05:49.186052 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:49.186036    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aeab8027-b217-476e-aa9f-bfc80ce02b93-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42\" (UID: \"aeab8027-b217-476e-aa9f-bfc80ce02b93\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42"
Sep 11 20:05:49.193750 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:49.193718    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb6c7\" (UniqueName: \"kubernetes.io/projected/aeab8027-b217-476e-aa9f-bfc80ce02b93-kube-api-access-vb6c7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42\" (UID: \"aeab8027-b217-476e-aa9f-bfc80ce02b93\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42"
Sep 11 20:05:49.304108 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:49.304019    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42"
Sep 11 20:05:49.573592 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:49.573566    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42"]
Sep 11 20:05:49.575675 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:05:49.575648    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeab8027_b217_476e_aa9f_bfc80ce02b93.slice/crio-ae4d6f6d9320eb219f2151be974cbd0172ee368de6b44fbfec4ac76272e2bdc6 WatchSource:0}: Error finding container ae4d6f6d9320eb219f2151be974cbd0172ee368de6b44fbfec4ac76272e2bdc6: Status 404 returned error can't find the container with id ae4d6f6d9320eb219f2151be974cbd0172ee368de6b44fbfec4ac76272e2bdc6
Sep 11 20:05:49.767737 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:49.767709    2580 generic.go:356] "Generic (PLEG): container finished" podID="aeab8027-b217-476e-aa9f-bfc80ce02b93" containerID="bffd48006fb40f6db0ccc7aa0acd1555e5970a2225580f6594a5b888ea0cfa25" exitCode=0
Sep 11 20:05:49.767874 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:49.767782    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42" event={"ID":"aeab8027-b217-476e-aa9f-bfc80ce02b93","Type":"ContainerDied","Data":"bffd48006fb40f6db0ccc7aa0acd1555e5970a2225580f6594a5b888ea0cfa25"}
Sep 11 20:05:49.767874 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:49.767819    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42" event={"ID":"aeab8027-b217-476e-aa9f-bfc80ce02b93","Type":"ContainerStarted","Data":"ae4d6f6d9320eb219f2151be974cbd0172ee368de6b44fbfec4ac76272e2bdc6"}
Sep 11 20:05:49.768244 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:49.768229    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 20:05:50.774019 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:50.773991    2580 generic.go:356] "Generic (PLEG): container finished" podID="aeab8027-b217-476e-aa9f-bfc80ce02b93" containerID="31647f68ab0cfed11780cf24a28a68113d36f69195c28542b67059481d1df55d" exitCode=0
Sep 11 20:05:50.774422 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:50.774079    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42" event={"ID":"aeab8027-b217-476e-aa9f-bfc80ce02b93","Type":"ContainerDied","Data":"31647f68ab0cfed11780cf24a28a68113d36f69195c28542b67059481d1df55d"}
Sep 11 20:05:51.780404 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:51.780372    2580 generic.go:356] "Generic (PLEG): container finished" podID="aeab8027-b217-476e-aa9f-bfc80ce02b93" containerID="5ea96aab5476b39be619618acd9f4312ec5db6e1b242098cbea3bf3591bef581" exitCode=0
Sep 11 20:05:51.780847 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:51.780456    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42" event={"ID":"aeab8027-b217-476e-aa9f-bfc80ce02b93","Type":"ContainerDied","Data":"5ea96aab5476b39be619618acd9f4312ec5db6e1b242098cbea3bf3591bef581"}
Sep 11 20:05:52.909203 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:52.909183    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42"
Sep 11 20:05:53.007936 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:53.007901    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aeab8027-b217-476e-aa9f-bfc80ce02b93-bundle\") pod \"aeab8027-b217-476e-aa9f-bfc80ce02b93\" (UID: \"aeab8027-b217-476e-aa9f-bfc80ce02b93\") "
Sep 11 20:05:53.008088 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:53.007944    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aeab8027-b217-476e-aa9f-bfc80ce02b93-util\") pod \"aeab8027-b217-476e-aa9f-bfc80ce02b93\" (UID: \"aeab8027-b217-476e-aa9f-bfc80ce02b93\") "
Sep 11 20:05:53.008088 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:53.007972    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb6c7\" (UniqueName: \"kubernetes.io/projected/aeab8027-b217-476e-aa9f-bfc80ce02b93-kube-api-access-vb6c7\") pod \"aeab8027-b217-476e-aa9f-bfc80ce02b93\" (UID: \"aeab8027-b217-476e-aa9f-bfc80ce02b93\") "
Sep 11 20:05:53.008643 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:53.008618    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeab8027-b217-476e-aa9f-bfc80ce02b93-bundle" (OuterVolumeSpecName: "bundle") pod "aeab8027-b217-476e-aa9f-bfc80ce02b93" (UID: "aeab8027-b217-476e-aa9f-bfc80ce02b93"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:05:53.009951 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:53.009925    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeab8027-b217-476e-aa9f-bfc80ce02b93-kube-api-access-vb6c7" (OuterVolumeSpecName: "kube-api-access-vb6c7") pod "aeab8027-b217-476e-aa9f-bfc80ce02b93" (UID: "aeab8027-b217-476e-aa9f-bfc80ce02b93"). InnerVolumeSpecName "kube-api-access-vb6c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:05:53.013327 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:53.013288    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeab8027-b217-476e-aa9f-bfc80ce02b93-util" (OuterVolumeSpecName: "util") pod "aeab8027-b217-476e-aa9f-bfc80ce02b93" (UID: "aeab8027-b217-476e-aa9f-bfc80ce02b93"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:05:53.108853 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:53.108786    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aeab8027-b217-476e-aa9f-bfc80ce02b93-bundle\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:05:53.108853 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:53.108811    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aeab8027-b217-476e-aa9f-bfc80ce02b93-util\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:05:53.108853 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:53.108820    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb6c7\" (UniqueName: \"kubernetes.io/projected/aeab8027-b217-476e-aa9f-bfc80ce02b93-kube-api-access-vb6c7\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:05:53.789955 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:53.789923    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42"
Sep 11 20:05:53.790115 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:53.789925    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnqc42" event={"ID":"aeab8027-b217-476e-aa9f-bfc80ce02b93","Type":"ContainerDied","Data":"ae4d6f6d9320eb219f2151be974cbd0172ee368de6b44fbfec4ac76272e2bdc6"}
Sep 11 20:05:53.790115 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:53.790046    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae4d6f6d9320eb219f2151be974cbd0172ee368de6b44fbfec4ac76272e2bdc6"
Sep 11 20:05:59.815609 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:59.815575    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw"]
Sep 11 20:05:59.816057 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:59.815838    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeab8027-b217-476e-aa9f-bfc80ce02b93" containerName="extract"
Sep 11 20:05:59.816057 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:59.815850    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeab8027-b217-476e-aa9f-bfc80ce02b93" containerName="extract"
Sep 11 20:05:59.816057 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:59.815858    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeab8027-b217-476e-aa9f-bfc80ce02b93" containerName="pull"
Sep 11 20:05:59.816057 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:59.815863    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeab8027-b217-476e-aa9f-bfc80ce02b93" containerName="pull"
Sep 11 20:05:59.816057 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:59.815870    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeab8027-b217-476e-aa9f-bfc80ce02b93" containerName="util"
Sep 11 20:05:59.816057 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:59.815875    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeab8027-b217-476e-aa9f-bfc80ce02b93" containerName="util"
Sep 11 20:05:59.816057 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:59.815937    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeab8027-b217-476e-aa9f-bfc80ce02b93" containerName="extract"
Sep 11 20:05:59.818003 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:59.817982    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw"
Sep 11 20:05:59.821912 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:59.821890    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 20:05:59.823566 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:59.823529    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-g5whd\""
Sep 11 20:05:59.823810 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:59.823791    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 20:05:59.834056 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:59.834032    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw"]
Sep 11 20:05:59.850543 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:59.850488    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d64f6b05-19b3-461d-b225-a8e2d6dc7b7b-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw\" (UID: \"d64f6b05-19b3-461d-b225-a8e2d6dc7b7b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw"
Sep 11 20:05:59.850707 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:59.850627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d64f6b05-19b3-461d-b225-a8e2d6dc7b7b-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw\" (UID: \"d64f6b05-19b3-461d-b225-a8e2d6dc7b7b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw"
Sep 11 20:05:59.850707 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:59.850694    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdnsb\" (UniqueName: \"kubernetes.io/projected/d64f6b05-19b3-461d-b225-a8e2d6dc7b7b-kube-api-access-tdnsb\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw\" (UID: \"d64f6b05-19b3-461d-b225-a8e2d6dc7b7b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw"
Sep 11 20:05:59.951429 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:59.951393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d64f6b05-19b3-461d-b225-a8e2d6dc7b7b-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw\" (UID: \"d64f6b05-19b3-461d-b225-a8e2d6dc7b7b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw"
Sep 11 20:05:59.951429 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:59.951428    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdnsb\" (UniqueName: \"kubernetes.io/projected/d64f6b05-19b3-461d-b225-a8e2d6dc7b7b-kube-api-access-tdnsb\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw\" (UID: \"d64f6b05-19b3-461d-b225-a8e2d6dc7b7b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw"
Sep 11 20:05:59.951645 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:59.951461    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d64f6b05-19b3-461d-b225-a8e2d6dc7b7b-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw\" (UID: \"d64f6b05-19b3-461d-b225-a8e2d6dc7b7b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw"
Sep 11 20:05:59.951861 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:59.951840    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d64f6b05-19b3-461d-b225-a8e2d6dc7b7b-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw\" (UID: \"d64f6b05-19b3-461d-b225-a8e2d6dc7b7b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw"
Sep 11 20:05:59.951897 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:59.951877    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d64f6b05-19b3-461d-b225-a8e2d6dc7b7b-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw\" (UID: \"d64f6b05-19b3-461d-b225-a8e2d6dc7b7b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw"
Sep 11 20:05:59.960106 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:05:59.960067    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdnsb\" (UniqueName: \"kubernetes.io/projected/d64f6b05-19b3-461d-b225-a8e2d6dc7b7b-kube-api-access-tdnsb\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw\" (UID: \"d64f6b05-19b3-461d-b225-a8e2d6dc7b7b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw"
Sep 11 20:06:00.126964 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:00.126878    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw"
Sep 11 20:06:00.412723 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:00.412698    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw"]
Sep 11 20:06:00.415179 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:06:00.415151    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64f6b05_19b3_461d_b225_a8e2d6dc7b7b.slice/crio-79779ad0a57c4365a38be290b1d94513c03754e22a39d6869b98aeb5efa03d69 WatchSource:0}: Error finding container 79779ad0a57c4365a38be290b1d94513c03754e22a39d6869b98aeb5efa03d69: Status 404 returned error can't find the container with id 79779ad0a57c4365a38be290b1d94513c03754e22a39d6869b98aeb5efa03d69
Sep 11 20:06:00.825756 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:00.825725    2580 generic.go:356] "Generic (PLEG): container finished" podID="d64f6b05-19b3-461d-b225-a8e2d6dc7b7b" containerID="f96afa3ea7d4a1cb9048681aadf7925e9e4780b1e3378d1ed5d89dd8da2800e4" exitCode=0
Sep 11 20:06:00.826138 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:00.825816    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw" event={"ID":"d64f6b05-19b3-461d-b225-a8e2d6dc7b7b","Type":"ContainerDied","Data":"f96afa3ea7d4a1cb9048681aadf7925e9e4780b1e3378d1ed5d89dd8da2800e4"}
Sep 11 20:06:00.826138 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:00.825889    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw" event={"ID":"d64f6b05-19b3-461d-b225-a8e2d6dc7b7b","Type":"ContainerStarted","Data":"79779ad0a57c4365a38be290b1d94513c03754e22a39d6869b98aeb5efa03d69"}
Sep 11 20:06:01.832232 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:01.832199    2580 generic.go:356] "Generic (PLEG): container finished" podID="d64f6b05-19b3-461d-b225-a8e2d6dc7b7b" containerID="4598b29cded755dc08751cd6b4e1230b3076c24df0ea39a7d90914bcddd61885" exitCode=0
Sep 11 20:06:01.832637 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:01.832261    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw" event={"ID":"d64f6b05-19b3-461d-b225-a8e2d6dc7b7b","Type":"ContainerDied","Data":"4598b29cded755dc08751cd6b4e1230b3076c24df0ea39a7d90914bcddd61885"}
Sep 11 20:06:01.930121 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:01.930076    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt"]
Sep 11 20:06:01.932677 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:01.932648    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt"
Sep 11 20:06:01.935772 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:01.935747    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 20:06:01.935924 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:01.935747    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jwddf\""
Sep 11 20:06:01.936138 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:01.936124    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 20:06:01.936220 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:01.936175    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 20:06:01.936459 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:01.936444    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 20:06:01.962029 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:01.961998    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt"]
Sep 11 20:06:01.967096 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:01.967064    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk845\" (UniqueName: \"kubernetes.io/projected/4bb46042-91b3-4506-857f-4db20fa98890-kube-api-access-nk845\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-24xlt\" (UID: \"4bb46042-91b3-4506-857f-4db20fa98890\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt"
Sep 11 20:06:01.967328 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:01.967301    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4bb46042-91b3-4506-857f-4db20fa98890-manifests\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-24xlt\" (UID: \"4bb46042-91b3-4506-857f-4db20fa98890\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt"
Sep 11 20:06:01.967448 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:01.967343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4bb46042-91b3-4506-857f-4db20fa98890-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-24xlt\" (UID: \"4bb46042-91b3-4506-857f-4db20fa98890\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt"
Sep 11 20:06:01.967448 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:01.967397    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4bb46042-91b3-4506-857f-4db20fa98890-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-24xlt\" (UID: \"4bb46042-91b3-4506-857f-4db20fa98890\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt"
Sep 11 20:06:01.967448 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:01.967439    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bb46042-91b3-4506-857f-4db20fa98890-tmp\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-24xlt\" (UID: \"4bb46042-91b3-4506-857f-4db20fa98890\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt"
Sep 11 20:06:02.067918 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:02.067870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk845\" (UniqueName: \"kubernetes.io/projected/4bb46042-91b3-4506-857f-4db20fa98890-kube-api-access-nk845\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-24xlt\" (UID: \"4bb46042-91b3-4506-857f-4db20fa98890\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt"
Sep 11 20:06:02.068137 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:02.067929    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4bb46042-91b3-4506-857f-4db20fa98890-manifests\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-24xlt\" (UID: \"4bb46042-91b3-4506-857f-4db20fa98890\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt"
Sep 11 20:06:02.068137 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:02.067951    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4bb46042-91b3-4506-857f-4db20fa98890-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-24xlt\" (UID: \"4bb46042-91b3-4506-857f-4db20fa98890\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt"
Sep 11 20:06:02.068137 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:02.067978    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4bb46042-91b3-4506-857f-4db20fa98890-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-24xlt\" (UID: \"4bb46042-91b3-4506-857f-4db20fa98890\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt"
Sep 11 20:06:02.068137 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:02.068009    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bb46042-91b3-4506-857f-4db20fa98890-tmp\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-24xlt\" (UID: \"4bb46042-91b3-4506-857f-4db20fa98890\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt"
Sep 11 20:06:02.068457 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:02.068425    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4bb46042-91b3-4506-857f-4db20fa98890-manifests\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-24xlt\" (UID: \"4bb46042-91b3-4506-857f-4db20fa98890\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt"
Sep 11 20:06:02.070460 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:02.070429    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bb46042-91b3-4506-857f-4db20fa98890-tmp\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-24xlt\" (UID: \"4bb46042-91b3-4506-857f-4db20fa98890\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt"
Sep 11 20:06:02.070704 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:02.070685    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4bb46042-91b3-4506-857f-4db20fa98890-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-24xlt\" (UID: \"4bb46042-91b3-4506-857f-4db20fa98890\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt"
Sep 11 20:06:02.070936 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:02.070915    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4bb46042-91b3-4506-857f-4db20fa98890-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-24xlt\" (UID: \"4bb46042-91b3-4506-857f-4db20fa98890\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt"
Sep 11 20:06:02.082053 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:02.082026    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk845\" (UniqueName: \"kubernetes.io/projected/4bb46042-91b3-4506-857f-4db20fa98890-kube-api-access-nk845\") pod \"opendatahub-operator-controller-manager-5fdf48ff49-24xlt\" (UID: \"4bb46042-91b3-4506-857f-4db20fa98890\") " pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt"
Sep 11 20:06:02.244823 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:02.244787    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt"
Sep 11 20:06:02.525109 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:02.525075    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt"]
Sep 11 20:06:02.526505 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:06:02.526459    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb46042_91b3_4506_857f_4db20fa98890.slice/crio-07012191e00fb22ee0fc682fba2320054aeef7ee677f32ecc51e6bc4c77fee98 WatchSource:0}: Error finding container 07012191e00fb22ee0fc682fba2320054aeef7ee677f32ecc51e6bc4c77fee98: Status 404 returned error can't find the container with id 07012191e00fb22ee0fc682fba2320054aeef7ee677f32ecc51e6bc4c77fee98
Sep 11 20:06:02.839010 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:02.838922    2580 generic.go:356] "Generic (PLEG): container finished" podID="d64f6b05-19b3-461d-b225-a8e2d6dc7b7b" containerID="5a32323b4ee1a9358a9ca01c13fa143be61722ab96ff7ff58b7adb1d0fe117bc" exitCode=0
Sep 11 20:06:02.839391 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:02.839011    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw" event={"ID":"d64f6b05-19b3-461d-b225-a8e2d6dc7b7b","Type":"ContainerDied","Data":"5a32323b4ee1a9358a9ca01c13fa143be61722ab96ff7ff58b7adb1d0fe117bc"}
Sep 11 20:06:02.840295 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:02.840273    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt" event={"ID":"4bb46042-91b3-4506-857f-4db20fa98890","Type":"ContainerStarted","Data":"07012191e00fb22ee0fc682fba2320054aeef7ee677f32ecc51e6bc4c77fee98"}
Sep 11 20:06:04.682317 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:04.682289    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw"
Sep 11 20:06:04.688358 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:04.688333    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d64f6b05-19b3-461d-b225-a8e2d6dc7b7b-util\") pod \"d64f6b05-19b3-461d-b225-a8e2d6dc7b7b\" (UID: \"d64f6b05-19b3-461d-b225-a8e2d6dc7b7b\") "
Sep 11 20:06:04.688539 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:04.688378    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdnsb\" (UniqueName: \"kubernetes.io/projected/d64f6b05-19b3-461d-b225-a8e2d6dc7b7b-kube-api-access-tdnsb\") pod \"d64f6b05-19b3-461d-b225-a8e2d6dc7b7b\" (UID: \"d64f6b05-19b3-461d-b225-a8e2d6dc7b7b\") "
Sep 11 20:06:04.688539 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:04.688427    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d64f6b05-19b3-461d-b225-a8e2d6dc7b7b-bundle\") pod \"d64f6b05-19b3-461d-b225-a8e2d6dc7b7b\" (UID: \"d64f6b05-19b3-461d-b225-a8e2d6dc7b7b\") "
Sep 11 20:06:04.689189 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:04.689165    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64f6b05-19b3-461d-b225-a8e2d6dc7b7b-bundle" (OuterVolumeSpecName: "bundle") pod "d64f6b05-19b3-461d-b225-a8e2d6dc7b7b" (UID: "d64f6b05-19b3-461d-b225-a8e2d6dc7b7b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:06:04.690899 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:04.690851    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d64f6b05-19b3-461d-b225-a8e2d6dc7b7b-kube-api-access-tdnsb" (OuterVolumeSpecName: "kube-api-access-tdnsb") pod "d64f6b05-19b3-461d-b225-a8e2d6dc7b7b" (UID: "d64f6b05-19b3-461d-b225-a8e2d6dc7b7b"). InnerVolumeSpecName "kube-api-access-tdnsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:06:04.694374 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:04.694322    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d64f6b05-19b3-461d-b225-a8e2d6dc7b7b-util" (OuterVolumeSpecName: "util") pod "d64f6b05-19b3-461d-b225-a8e2d6dc7b7b" (UID: "d64f6b05-19b3-461d-b225-a8e2d6dc7b7b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:06:04.789184 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:04.789149    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d64f6b05-19b3-461d-b225-a8e2d6dc7b7b-bundle\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:06:04.789184 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:04.789181    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d64f6b05-19b3-461d-b225-a8e2d6dc7b7b-util\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:06:04.789184 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:04.789194    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdnsb\" (UniqueName: \"kubernetes.io/projected/d64f6b05-19b3-461d-b225-a8e2d6dc7b7b-kube-api-access-tdnsb\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:06:04.856611 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:04.856574    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw"
Sep 11 20:06:04.856760 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:04.856574    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05rbxw" event={"ID":"d64f6b05-19b3-461d-b225-a8e2d6dc7b7b","Type":"ContainerDied","Data":"79779ad0a57c4365a38be290b1d94513c03754e22a39d6869b98aeb5efa03d69"}
Sep 11 20:06:04.856813 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:04.856762    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79779ad0a57c4365a38be290b1d94513c03754e22a39d6869b98aeb5efa03d69"
Sep 11 20:06:05.863110 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:05.863029    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt" event={"ID":"4bb46042-91b3-4506-857f-4db20fa98890","Type":"ContainerStarted","Data":"5c8701f0ab24e884cbb7a8248dcb07ff6879396f1250ddb2fbc8e4cd508b32be"}
Sep 11 20:06:06.869113 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:06.869080    2580 generic.go:356] "Generic (PLEG): container finished" podID="4bb46042-91b3-4506-857f-4db20fa98890" containerID="5c8701f0ab24e884cbb7a8248dcb07ff6879396f1250ddb2fbc8e4cd508b32be" exitCode=0
Sep 11 20:06:06.869483 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:06.869175    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt" event={"ID":"4bb46042-91b3-4506-857f-4db20fa98890","Type":"ContainerDied","Data":"5c8701f0ab24e884cbb7a8248dcb07ff6879396f1250ddb2fbc8e4cd508b32be"}
Sep 11 20:06:07.875137 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:07.875054    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt" event={"ID":"4bb46042-91b3-4506-857f-4db20fa98890","Type":"ContainerStarted","Data":"d2529a920e4d4639bea0da8bf2ed6818bb5ddc36b1390d33ebbacc571c3dd0d4"}
Sep 11 20:06:07.875471 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:07.875149    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt"
Sep 11 20:06:07.907279 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:07.907221    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt" podStartSLOduration=2.15684025 podStartE2EDuration="6.907204033s" totalImagesPullingTime="3.654476017s" totalInitContainerRuntime="1.095887766s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:06:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:06:07.904423789 +0000 UTC m=+337.896586058" watchObservedRunningTime="2026-09-11 20:06:07.907204033 +0000 UTC m=+337.899366303"
Sep 11 20:06:18.881340 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:18.881303    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5fdf48ff49-24xlt"
Sep 11 20:06:22.054614 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.054558    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-567ff9dcd5-hrxd6"]
Sep 11 20:06:22.055117 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.054960    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d64f6b05-19b3-461d-b225-a8e2d6dc7b7b" containerName="pull"
Sep 11 20:06:22.055117 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.054980    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d64f6b05-19b3-461d-b225-a8e2d6dc7b7b" containerName="pull"
Sep 11 20:06:22.055117 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.054991    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d64f6b05-19b3-461d-b225-a8e2d6dc7b7b" containerName="extract"
Sep 11 20:06:22.055117 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.054999    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d64f6b05-19b3-461d-b225-a8e2d6dc7b7b" containerName="extract"
Sep 11 20:06:22.055117 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.055019    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d64f6b05-19b3-461d-b225-a8e2d6dc7b7b" containerName="util"
Sep 11 20:06:22.055117 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.055027    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d64f6b05-19b3-461d-b225-a8e2d6dc7b7b" containerName="util"
Sep 11 20:06:22.055117 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.055102    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d64f6b05-19b3-461d-b225-a8e2d6dc7b7b" containerName="extract"
Sep 11 20:06:22.064642 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.064619    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-hrxd6"
Sep 11 20:06:22.067713 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.067665    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 20:06:22.068167 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.068145    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 20:06:22.068275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.068253    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 20:06:22.068341 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.068295    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 20:06:22.068865 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.068849    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-wjqrv\""
Sep 11 20:06:22.087616 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.087591    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-567ff9dcd5-hrxd6"]
Sep 11 20:06:22.111887 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.111853    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/956f104b-830b-4d30-8185-d446debf54c3-tls-certs\") pod \"kube-auth-proxy-567ff9dcd5-hrxd6\" (UID: \"956f104b-830b-4d30-8185-d446debf54c3\") " pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-hrxd6"
Sep 11 20:06:22.112024 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.111897    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/956f104b-830b-4d30-8185-d446debf54c3-tmp\") pod \"kube-auth-proxy-567ff9dcd5-hrxd6\" (UID: \"956f104b-830b-4d30-8185-d446debf54c3\") " pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-hrxd6"
Sep 11 20:06:22.112024 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.111921    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gvvl\" (UniqueName: \"kubernetes.io/projected/956f104b-830b-4d30-8185-d446debf54c3-kube-api-access-6gvvl\") pod \"kube-auth-proxy-567ff9dcd5-hrxd6\" (UID: \"956f104b-830b-4d30-8185-d446debf54c3\") " pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-hrxd6"
Sep 11 20:06:22.213226 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.213168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/956f104b-830b-4d30-8185-d446debf54c3-tmp\") pod \"kube-auth-proxy-567ff9dcd5-hrxd6\" (UID: \"956f104b-830b-4d30-8185-d446debf54c3\") " pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-hrxd6"
Sep 11 20:06:22.213226 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.213217    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gvvl\" (UniqueName: \"kubernetes.io/projected/956f104b-830b-4d30-8185-d446debf54c3-kube-api-access-6gvvl\") pod \"kube-auth-proxy-567ff9dcd5-hrxd6\" (UID: \"956f104b-830b-4d30-8185-d446debf54c3\") " pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-hrxd6"
Sep 11 20:06:22.213576 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.213259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/956f104b-830b-4d30-8185-d446debf54c3-tls-certs\") pod \"kube-auth-proxy-567ff9dcd5-hrxd6\" (UID: \"956f104b-830b-4d30-8185-d446debf54c3\") " pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-hrxd6"
Sep 11 20:06:22.215605 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.215576    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/956f104b-830b-4d30-8185-d446debf54c3-tmp\") pod \"kube-auth-proxy-567ff9dcd5-hrxd6\" (UID: \"956f104b-830b-4d30-8185-d446debf54c3\") " pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-hrxd6"
Sep 11 20:06:22.215802 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.215785    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/956f104b-830b-4d30-8185-d446debf54c3-tls-certs\") pod \"kube-auth-proxy-567ff9dcd5-hrxd6\" (UID: \"956f104b-830b-4d30-8185-d446debf54c3\") " pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-hrxd6"
Sep 11 20:06:22.224779 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.224750    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gvvl\" (UniqueName: \"kubernetes.io/projected/956f104b-830b-4d30-8185-d446debf54c3-kube-api-access-6gvvl\") pod \"kube-auth-proxy-567ff9dcd5-hrxd6\" (UID: \"956f104b-830b-4d30-8185-d446debf54c3\") " pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-hrxd6"
Sep 11 20:06:22.374153 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.374062    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-hrxd6"
Sep 11 20:06:22.645744 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.645665    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-567ff9dcd5-hrxd6"]
Sep 11 20:06:22.649022 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:06:22.648985    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod956f104b_830b_4d30_8185_d446debf54c3.slice/crio-5d25a1f684b6dbbd67241a506e42510f92e31154e530dd8cf72a2acaa67a211e WatchSource:0}: Error finding container 5d25a1f684b6dbbd67241a506e42510f92e31154e530dd8cf72a2acaa67a211e: Status 404 returned error can't find the container with id 5d25a1f684b6dbbd67241a506e42510f92e31154e530dd8cf72a2acaa67a211e
Sep 11 20:06:22.940290 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:22.940259    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-hrxd6" event={"ID":"956f104b-830b-4d30-8185-d446debf54c3","Type":"ContainerStarted","Data":"5d25a1f684b6dbbd67241a506e42510f92e31154e530dd8cf72a2acaa67a211e"}
Sep 11 20:06:23.429586 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.429505    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"]
Sep 11 20:06:23.432946 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.432919    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:23.436007 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.435985    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 11 20:06:23.436148 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.436017    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 11 20:06:23.436148 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.436018    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 20:06:23.436264 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.436245    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 11 20:06:23.436743 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.436724    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gr65x\""
Sep 11 20:06:23.449533 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.449462    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"]
Sep 11 20:06:23.523810 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.523753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-th752\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:23.524059 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.523803    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-th752\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:23.524200 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.524125    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-th752\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:23.524200 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.524167    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-th752\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:23.524316 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.524215    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xwp8\" (UniqueName: \"kubernetes.io/projected/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-kube-api-access-4xwp8\") pod \"istiod-openshift-gateway-7b7995d5cd-th752\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:23.524316 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.524268    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-th752\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:23.524425 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.524366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-th752\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:23.625026 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.624987    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-th752\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:23.625206 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.625041    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-th752\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:23.625206 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.625067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-th752\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:23.625206 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.625117    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-th752\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:23.625206 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.625151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-th752\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:23.625206 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.625185    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xwp8\" (UniqueName: \"kubernetes.io/projected/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-kube-api-access-4xwp8\") pod \"istiod-openshift-gateway-7b7995d5cd-th752\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:23.625471 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.625210    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-th752\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:23.626382 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.626352    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-th752\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:23.627745 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.627721    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-th752\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:23.627856 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.627776    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-th752\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:23.628274 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.628234    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-th752\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:23.628433 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.628411    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-th752\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:23.633873 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.633845    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xwp8\" (UniqueName: \"kubernetes.io/projected/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-kube-api-access-4xwp8\") pod \"istiod-openshift-gateway-7b7995d5cd-th752\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:23.634007 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.633987    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-th752\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:23.744587 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:23.744547    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:24.030973 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:24.030944    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"]
Sep 11 20:06:24.032528 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:06:24.032470    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9c5ac0_e935_4de2_9a02_8397f7e5e2d8.slice/crio-727f4d9c7e70adb38423eec1962b13eb10e9cb16a948009f24690b7c48177586 WatchSource:0}: Error finding container 727f4d9c7e70adb38423eec1962b13eb10e9cb16a948009f24690b7c48177586: Status 404 returned error can't find the container with id 727f4d9c7e70adb38423eec1962b13eb10e9cb16a948009f24690b7c48177586
Sep 11 20:06:24.953931 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:24.953888    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752" event={"ID":"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8","Type":"ContainerStarted","Data":"727f4d9c7e70adb38423eec1962b13eb10e9cb16a948009f24690b7c48177586"}
Sep 11 20:06:25.962279 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:25.962227    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-hrxd6" event={"ID":"956f104b-830b-4d30-8185-d446debf54c3","Type":"ContainerStarted","Data":"835566e891b2ad6f10cdaa1c2327c6c710f754c1b031fb20ae9ff5b72944db0a"}
Sep 11 20:06:25.982483 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:25.982427    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-567ff9dcd5-hrxd6" podStartSLOduration=1.063103257 podStartE2EDuration="3.982408598s" totalImagesPullingTime="2.919305341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:06:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:06:25.98107717 +0000 UTC m=+355.973239442" watchObservedRunningTime="2026-09-11 20:06:25.982408598 +0000 UTC m=+355.974570870"
Sep 11 20:06:27.425062 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:27.425020    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 20:06:27.425401 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:27.425106    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 20:06:27.973202 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:27.973163    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752" event={"ID":"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8","Type":"ContainerStarted","Data":"fc9b8898a408940c658ad12228d835a6e58cecbeb1c6c9b4b239e8836fc4dc5d"}
Sep 11 20:06:27.973409 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:27.973257    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:27.999766 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:27.999712    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752" podStartSLOduration=1.6100826179999999 podStartE2EDuration="4.999694491s" totalImagesPullingTime="3.389611873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:06:27.998019206 +0000 UTC m=+357.990181477" watchObservedRunningTime="2026-09-11 20:06:27.999694491 +0000 UTC m=+357.991856761"
Sep 11 20:06:28.978934 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:28.978907    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:40.732970 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:40.732925    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-865445d586-svn2l"]
Sep 11 20:06:40.737951 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:40.737927    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-865445d586-svn2l"
Sep 11 20:06:40.741864 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:40.741837    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 20:06:40.741999 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:40.741837    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 20:06:40.742091 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:40.742062    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 20:06:40.742149 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:40.742096    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 20:06:40.742149 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:40.742096    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-x829v\""
Sep 11 20:06:40.742247 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:40.742205    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 20:06:40.747900 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:40.747877    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-865445d586-svn2l"]
Sep 11 20:06:40.878474 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:40.878425    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/638ffad9-e877-4c31-8975-aeede2e0b352-cert\") pod \"lws-controller-manager-865445d586-svn2l\" (UID: \"638ffad9-e877-4c31-8975-aeede2e0b352\") " pod="openshift-lws-operator/lws-controller-manager-865445d586-svn2l"
Sep 11 20:06:40.878687 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:40.878535    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/638ffad9-e877-4c31-8975-aeede2e0b352-manager-config\") pod \"lws-controller-manager-865445d586-svn2l\" (UID: \"638ffad9-e877-4c31-8975-aeede2e0b352\") " pod="openshift-lws-operator/lws-controller-manager-865445d586-svn2l"
Sep 11 20:06:40.878687 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:40.878594    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr7bl\" (UniqueName: \"kubernetes.io/projected/638ffad9-e877-4c31-8975-aeede2e0b352-kube-api-access-rr7bl\") pod \"lws-controller-manager-865445d586-svn2l\" (UID: \"638ffad9-e877-4c31-8975-aeede2e0b352\") " pod="openshift-lws-operator/lws-controller-manager-865445d586-svn2l"
Sep 11 20:06:40.878687 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:40.878669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/638ffad9-e877-4c31-8975-aeede2e0b352-metrics-cert\") pod \"lws-controller-manager-865445d586-svn2l\" (UID: \"638ffad9-e877-4c31-8975-aeede2e0b352\") " pod="openshift-lws-operator/lws-controller-manager-865445d586-svn2l"
Sep 11 20:06:40.979681 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:40.979644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/638ffad9-e877-4c31-8975-aeede2e0b352-metrics-cert\") pod \"lws-controller-manager-865445d586-svn2l\" (UID: \"638ffad9-e877-4c31-8975-aeede2e0b352\") " pod="openshift-lws-operator/lws-controller-manager-865445d586-svn2l"
Sep 11 20:06:40.979858 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:40.979735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/638ffad9-e877-4c31-8975-aeede2e0b352-cert\") pod \"lws-controller-manager-865445d586-svn2l\" (UID: \"638ffad9-e877-4c31-8975-aeede2e0b352\") " pod="openshift-lws-operator/lws-controller-manager-865445d586-svn2l"
Sep 11 20:06:40.979858 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:40.979752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/638ffad9-e877-4c31-8975-aeede2e0b352-manager-config\") pod \"lws-controller-manager-865445d586-svn2l\" (UID: \"638ffad9-e877-4c31-8975-aeede2e0b352\") " pod="openshift-lws-operator/lws-controller-manager-865445d586-svn2l"
Sep 11 20:06:40.979858 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:40.979767    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr7bl\" (UniqueName: \"kubernetes.io/projected/638ffad9-e877-4c31-8975-aeede2e0b352-kube-api-access-rr7bl\") pod \"lws-controller-manager-865445d586-svn2l\" (UID: \"638ffad9-e877-4c31-8975-aeede2e0b352\") " pod="openshift-lws-operator/lws-controller-manager-865445d586-svn2l"
Sep 11 20:06:40.980564 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:40.980534    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/638ffad9-e877-4c31-8975-aeede2e0b352-manager-config\") pod \"lws-controller-manager-865445d586-svn2l\" (UID: \"638ffad9-e877-4c31-8975-aeede2e0b352\") " pod="openshift-lws-operator/lws-controller-manager-865445d586-svn2l"
Sep 11 20:06:40.982577 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:40.982551    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/638ffad9-e877-4c31-8975-aeede2e0b352-metrics-cert\") pod \"lws-controller-manager-865445d586-svn2l\" (UID: \"638ffad9-e877-4c31-8975-aeede2e0b352\") " pod="openshift-lws-operator/lws-controller-manager-865445d586-svn2l"
Sep 11 20:06:40.982577 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:40.982560    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/638ffad9-e877-4c31-8975-aeede2e0b352-cert\") pod \"lws-controller-manager-865445d586-svn2l\" (UID: \"638ffad9-e877-4c31-8975-aeede2e0b352\") " pod="openshift-lws-operator/lws-controller-manager-865445d586-svn2l"
Sep 11 20:06:40.991605 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:40.991560    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr7bl\" (UniqueName: \"kubernetes.io/projected/638ffad9-e877-4c31-8975-aeede2e0b352-kube-api-access-rr7bl\") pod \"lws-controller-manager-865445d586-svn2l\" (UID: \"638ffad9-e877-4c31-8975-aeede2e0b352\") " pod="openshift-lws-operator/lws-controller-manager-865445d586-svn2l"
Sep 11 20:06:41.047682 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:41.047652    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-865445d586-svn2l"
Sep 11 20:06:41.327251 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:41.327227    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-865445d586-svn2l"]
Sep 11 20:06:41.329562 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:06:41.329536    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638ffad9_e877_4c31_8975_aeede2e0b352.slice/crio-554425e5eeb7dd5911c0c0513e8585bd4547c0da46dd102fbbc14fdbc4c9040c WatchSource:0}: Error finding container 554425e5eeb7dd5911c0c0513e8585bd4547c0da46dd102fbbc14fdbc4c9040c: Status 404 returned error can't find the container with id 554425e5eeb7dd5911c0c0513e8585bd4547c0da46dd102fbbc14fdbc4c9040c
Sep 11 20:06:42.036280 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:42.036242    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-865445d586-svn2l" event={"ID":"638ffad9-e877-4c31-8975-aeede2e0b352","Type":"ContainerStarted","Data":"554425e5eeb7dd5911c0c0513e8585bd4547c0da46dd102fbbc14fdbc4c9040c"}
Sep 11 20:06:44.047814 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.047763    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-865445d586-svn2l" event={"ID":"638ffad9-e877-4c31-8975-aeede2e0b352","Type":"ContainerStarted","Data":"2262e9fd5b3f4fedc042082af21577c8e0c8270671a49650cda7af07ecfc7ccf"}
Sep 11 20:06:44.048217 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.047868    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-865445d586-svn2l"
Sep 11 20:06:44.082119 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.082069    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-865445d586-svn2l" podStartSLOduration=2.053256383 podStartE2EDuration="4.08205254s" totalImagesPullingTime="2.028796157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:06:44.080797766 +0000 UTC m=+374.072960035" watchObservedRunningTime="2026-09-11 20:06:44.08205254 +0000 UTC m=+374.074214810"
Sep 11 20:06:44.172109 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.172072    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"]
Sep 11 20:06:44.172288 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.172262    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752" podUID="9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8" containerName="discovery" containerID="cri-o://fc9b8898a408940c658ad12228d835a6e58cecbeb1c6c9b4b239e8836fc4dc5d" gracePeriod=30
Sep 11 20:06:44.408217 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.408193    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:44.511559 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.511529    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-local-certs\") pod \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") "
Sep 11 20:06:44.511559 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.511561    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-istio-token\") pod \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") "
Sep 11 20:06:44.511751 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.511607    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-cacerts\") pod \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") "
Sep 11 20:06:44.511751 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.511635    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-istio-kubeconfig\") pod \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") "
Sep 11 20:06:44.511751 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.511663    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-istio-csr-dns-cert\") pod \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") "
Sep 11 20:06:44.511751 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.511689    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xwp8\" (UniqueName: \"kubernetes.io/projected/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-kube-api-access-4xwp8\") pod \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") "
Sep 11 20:06:44.511751 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.511747    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-istio-csr-ca-configmap\") pod \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\" (UID: \"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8\") "
Sep 11 20:06:44.512434 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.512403    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8" (UID: "9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:06:44.514019 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.513983    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8" (UID: "9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:06:44.514205 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.514176    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-istio-token" (OuterVolumeSpecName: "istio-token") pod "9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8" (UID: "9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:06:44.514205 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.514175    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-local-certs" (OuterVolumeSpecName: "local-certs") pod "9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8" (UID: "9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:06:44.514323 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.514221    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-cacerts" (OuterVolumeSpecName: "cacerts") pod "9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8" (UID: "9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:06:44.514406 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.514385    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8" (UID: "9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:06:44.514466 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.514413    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-kube-api-access-4xwp8" (OuterVolumeSpecName: "kube-api-access-4xwp8") pod "9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8" (UID: "9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8"). InnerVolumeSpecName "kube-api-access-4xwp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:06:44.613010 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.612948    2580 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-cacerts\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:06:44.613010 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.612971    2580 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-istio-kubeconfig\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:06:44.613010 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.612982    2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-istio-csr-dns-cert\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:06:44.613010 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.612991    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xwp8\" (UniqueName: \"kubernetes.io/projected/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-kube-api-access-4xwp8\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:06:44.613010 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.612999    2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-istio-csr-ca-configmap\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:06:44.613010 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.613007    2580 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-local-certs\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:06:44.613260 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:44.613016    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8-istio-token\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:06:45.053448 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:45.053411    2580 generic.go:356] "Generic (PLEG): container finished" podID="9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8" containerID="fc9b8898a408940c658ad12228d835a6e58cecbeb1c6c9b4b239e8836fc4dc5d" exitCode=0
Sep 11 20:06:45.053839 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:45.053507    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"
Sep 11 20:06:45.053839 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:45.053510    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752" event={"ID":"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8","Type":"ContainerDied","Data":"fc9b8898a408940c658ad12228d835a6e58cecbeb1c6c9b4b239e8836fc4dc5d"}
Sep 11 20:06:45.053839 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:45.053589    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752" event={"ID":"9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8","Type":"ContainerDied","Data":"727f4d9c7e70adb38423eec1962b13eb10e9cb16a948009f24690b7c48177586"}
Sep 11 20:06:45.053839 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:45.053614    2580 scope.go:117] "RemoveContainer" containerID="fc9b8898a408940c658ad12228d835a6e58cecbeb1c6c9b4b239e8836fc4dc5d"
Sep 11 20:06:45.062330 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:45.062313    2580 scope.go:117] "RemoveContainer" containerID="fc9b8898a408940c658ad12228d835a6e58cecbeb1c6c9b4b239e8836fc4dc5d"
Sep 11 20:06:45.062598 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:06:45.062579    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc9b8898a408940c658ad12228d835a6e58cecbeb1c6c9b4b239e8836fc4dc5d\": container with ID starting with fc9b8898a408940c658ad12228d835a6e58cecbeb1c6c9b4b239e8836fc4dc5d not found: ID does not exist" containerID="fc9b8898a408940c658ad12228d835a6e58cecbeb1c6c9b4b239e8836fc4dc5d"
Sep 11 20:06:45.062668 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:45.062604    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc9b8898a408940c658ad12228d835a6e58cecbeb1c6c9b4b239e8836fc4dc5d"} err="failed to get container status \"fc9b8898a408940c658ad12228d835a6e58cecbeb1c6c9b4b239e8836fc4dc5d\": rpc error: code = NotFound desc = could not find container \"fc9b8898a408940c658ad12228d835a6e58cecbeb1c6c9b4b239e8836fc4dc5d\": container with ID starting with fc9b8898a408940c658ad12228d835a6e58cecbeb1c6c9b4b239e8836fc4dc5d not found: ID does not exist"
Sep 11 20:06:45.081728 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:45.081706    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"]
Sep 11 20:06:45.090059 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:45.090039    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-th752"]
Sep 11 20:06:46.440487 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:46.440451    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8" path="/var/lib/kubelet/pods/9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8/volumes"
Sep 11 20:06:49.189457 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:49.189422    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cf8bb5bb-6ht2v"]
Sep 11 20:06:55.055899 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:06:55.055865    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-865445d586-svn2l"
Sep 11 20:07:14.208705 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:14.208643    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6cf8bb5bb-6ht2v" podUID="5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4" containerName="console" containerID="cri-o://91e15a8b57b39428d9c9981da59b16839921fdb83c4049eae6dc7a2da62fd5b7" gracePeriod=15
Sep 11 20:07:14.444383 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:14.444359    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:07:14.456314 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:14.456295    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-service-ca\") pod \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") "
Sep 11 20:07:14.456407 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:14.456330    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-console-oauth-config\") pod \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") "
Sep 11 20:07:14.456407 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:14.456357    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-console-serving-cert\") pod \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") "
Sep 11 20:07:14.456407 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:14.456372    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-oauth-serving-cert\") pod \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") "
Sep 11 20:07:14.456579 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:14.456504    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnmbx\" (UniqueName: \"kubernetes.io/projected/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-kube-api-access-dnmbx\") pod \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") "
Sep 11 20:07:14.456579 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:14.456549    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-trusted-ca-bundle\") pod \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") "
Sep 11 20:07:14.456683 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:14.456587    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-console-config\") pod \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\" (UID: \"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4\") "
Sep 11 20:07:14.456734 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:14.456679    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-service-ca" (OuterVolumeSpecName: "service-ca") pod "5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4" (UID: "5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:07:14.456869 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:14.456844    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4" (UID: "5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:07:14.457009 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:14.456933    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4" (UID: "5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:07:14.457009 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:14.456940    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-console-config" (OuterVolumeSpecName: "console-config") pod "5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4" (UID: "5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:07:14.457149 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:14.456960    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-service-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:07:14.458459 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:14.458433    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4" (UID: "5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:07:14.458561 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:14.458484    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4" (UID: "5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:07:14.458652 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:14.458626    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-kube-api-access-dnmbx" (OuterVolumeSpecName: "kube-api-access-dnmbx") pod "5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4" (UID: "5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4"). InnerVolumeSpecName "kube-api-access-dnmbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:07:14.557795 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:14.557761    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-trusted-ca-bundle\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:07:14.557795 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:14.557788    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-console-config\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:07:14.557795 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:14.557797    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-console-oauth-config\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:07:14.558006 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:14.557806    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-console-serving-cert\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:07:14.558006 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:14.557815    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-oauth-serving-cert\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:07:14.558006 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:14.557823    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnmbx\" (UniqueName: \"kubernetes.io/projected/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4-kube-api-access-dnmbx\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:07:15.189048 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:15.189016    2580 generic.go:356] "Generic (PLEG): container finished" podID="5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4" containerID="91e15a8b57b39428d9c9981da59b16839921fdb83c4049eae6dc7a2da62fd5b7" exitCode=0
Sep 11 20:07:15.189242 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:15.189057    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf8bb5bb-6ht2v"
Sep 11 20:07:15.189242 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:15.189068    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf8bb5bb-6ht2v" event={"ID":"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4","Type":"ContainerDied","Data":"91e15a8b57b39428d9c9981da59b16839921fdb83c4049eae6dc7a2da62fd5b7"}
Sep 11 20:07:15.189242 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:15.189116    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf8bb5bb-6ht2v" event={"ID":"5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4","Type":"ContainerDied","Data":"e24f3a15695ee1fa77d1d8f548784ccccd4173b59bd73c2358bf8386c1548f24"}
Sep 11 20:07:15.189242 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:15.189141    2580 scope.go:117] "RemoveContainer" containerID="91e15a8b57b39428d9c9981da59b16839921fdb83c4049eae6dc7a2da62fd5b7"
Sep 11 20:07:15.197798 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:15.197781    2580 scope.go:117] "RemoveContainer" containerID="91e15a8b57b39428d9c9981da59b16839921fdb83c4049eae6dc7a2da62fd5b7"
Sep 11 20:07:15.198060 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:07:15.198041    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e15a8b57b39428d9c9981da59b16839921fdb83c4049eae6dc7a2da62fd5b7\": container with ID starting with 91e15a8b57b39428d9c9981da59b16839921fdb83c4049eae6dc7a2da62fd5b7 not found: ID does not exist" containerID="91e15a8b57b39428d9c9981da59b16839921fdb83c4049eae6dc7a2da62fd5b7"
Sep 11 20:07:15.198106 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:15.198069    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e15a8b57b39428d9c9981da59b16839921fdb83c4049eae6dc7a2da62fd5b7"} err="failed to get container status \"91e15a8b57b39428d9c9981da59b16839921fdb83c4049eae6dc7a2da62fd5b7\": rpc error: code = NotFound desc = could not find container \"91e15a8b57b39428d9c9981da59b16839921fdb83c4049eae6dc7a2da62fd5b7\": container with ID starting with 91e15a8b57b39428d9c9981da59b16839921fdb83c4049eae6dc7a2da62fd5b7 not found: ID does not exist"
Sep 11 20:07:15.210055 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:15.210025    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cf8bb5bb-6ht2v"]
Sep 11 20:07:15.215487 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:15.215468    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cf8bb5bb-6ht2v"]
Sep 11 20:07:16.440002 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:16.439970    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4" path="/var/lib/kubelet/pods/5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4/volumes"
Sep 11 20:07:37.614608 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.614572    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9"]
Sep 11 20:07:37.614987 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.614860    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4" containerName="console"
Sep 11 20:07:37.614987 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.614872    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4" containerName="console"
Sep 11 20:07:37.614987 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.614885    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8" containerName="discovery"
Sep 11 20:07:37.614987 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.614891    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8" containerName="discovery"
Sep 11 20:07:37.614987 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.614954    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b54c5d7-07cf-4ee1-9903-77cd6ac42fc4" containerName="console"
Sep 11 20:07:37.614987 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.614961    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a9c5ac0-e935-4de2-9a02-8397f7e5e2d8" containerName="discovery"
Sep 11 20:07:37.619165 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.619145    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9"
Sep 11 20:07:37.621601 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.621579    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 20:07:37.621702 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.621654    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-g5whd\""
Sep 11 20:07:37.622637 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.622623    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 20:07:37.625836 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.625642    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9"]
Sep 11 20:07:37.719131 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.719098    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4"]
Sep 11 20:07:37.722425 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.722409    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4"
Sep 11 20:07:37.723513 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.723473    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nk6f\" (UniqueName: \"kubernetes.io/projected/575db1d6-4eb0-4999-a540-a6051be7565f-kube-api-access-4nk6f\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9\" (UID: \"575db1d6-4eb0-4999-a540-a6051be7565f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9"
Sep 11 20:07:37.723593 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.723581    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/575db1d6-4eb0-4999-a540-a6051be7565f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9\" (UID: \"575db1d6-4eb0-4999-a540-a6051be7565f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9"
Sep 11 20:07:37.723638 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.723609    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/575db1d6-4eb0-4999-a540-a6051be7565f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9\" (UID: \"575db1d6-4eb0-4999-a540-a6051be7565f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9"
Sep 11 20:07:37.731109 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.731089    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4"]
Sep 11 20:07:37.817062 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.817029    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk"]
Sep 11 20:07:37.820346 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.820328    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk"
Sep 11 20:07:37.824104 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.824083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/575db1d6-4eb0-4999-a540-a6051be7565f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9\" (UID: \"575db1d6-4eb0-4999-a540-a6051be7565f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9"
Sep 11 20:07:37.824212 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.824117    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/575db1d6-4eb0-4999-a540-a6051be7565f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9\" (UID: \"575db1d6-4eb0-4999-a540-a6051be7565f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9"
Sep 11 20:07:37.824212 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.824167    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e5a047b-fa22-4bb2-8a4d-69e0d890eda1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4\" (UID: \"0e5a047b-fa22-4bb2-8a4d-69e0d890eda1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4"
Sep 11 20:07:37.824212 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.824192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6wlz\" (UniqueName: \"kubernetes.io/projected/0e5a047b-fa22-4bb2-8a4d-69e0d890eda1-kube-api-access-w6wlz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4\" (UID: \"0e5a047b-fa22-4bb2-8a4d-69e0d890eda1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4"
Sep 11 20:07:37.824343 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.824218    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nk6f\" (UniqueName: \"kubernetes.io/projected/575db1d6-4eb0-4999-a540-a6051be7565f-kube-api-access-4nk6f\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9\" (UID: \"575db1d6-4eb0-4999-a540-a6051be7565f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9"
Sep 11 20:07:37.824343 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.824236    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e5a047b-fa22-4bb2-8a4d-69e0d890eda1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4\" (UID: \"0e5a047b-fa22-4bb2-8a4d-69e0d890eda1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4"
Sep 11 20:07:37.824528 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.824484    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/575db1d6-4eb0-4999-a540-a6051be7565f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9\" (UID: \"575db1d6-4eb0-4999-a540-a6051be7565f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9"
Sep 11 20:07:37.824570 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.824544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/575db1d6-4eb0-4999-a540-a6051be7565f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9\" (UID: \"575db1d6-4eb0-4999-a540-a6051be7565f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9"
Sep 11 20:07:37.828741 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.828690    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk"]
Sep 11 20:07:37.832531 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.832506    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nk6f\" (UniqueName: \"kubernetes.io/projected/575db1d6-4eb0-4999-a540-a6051be7565f-kube-api-access-4nk6f\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9\" (UID: \"575db1d6-4eb0-4999-a540-a6051be7565f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9"
Sep 11 20:07:37.921459 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.921397    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx"]
Sep 11 20:07:37.924669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.924650    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx"
Sep 11 20:07:37.924890 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.924870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/328a0ef9-0b11-402a-b060-111e9e3593b4-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk\" (UID: \"328a0ef9-0b11-402a-b060-111e9e3593b4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk"
Sep 11 20:07:37.924945 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.924930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e5a047b-fa22-4bb2-8a4d-69e0d890eda1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4\" (UID: \"0e5a047b-fa22-4bb2-8a4d-69e0d890eda1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4"
Sep 11 20:07:37.924985 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.924967    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7hg9\" (UniqueName: \"kubernetes.io/projected/328a0ef9-0b11-402a-b060-111e9e3593b4-kube-api-access-s7hg9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk\" (UID: \"328a0ef9-0b11-402a-b060-111e9e3593b4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk"
Sep 11 20:07:37.925019 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.924997    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6wlz\" (UniqueName: \"kubernetes.io/projected/0e5a047b-fa22-4bb2-8a4d-69e0d890eda1-kube-api-access-w6wlz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4\" (UID: \"0e5a047b-fa22-4bb2-8a4d-69e0d890eda1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4"
Sep 11 20:07:37.925052 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.925021    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e5a047b-fa22-4bb2-8a4d-69e0d890eda1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4\" (UID: \"0e5a047b-fa22-4bb2-8a4d-69e0d890eda1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4"
Sep 11 20:07:37.925052 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.925042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/328a0ef9-0b11-402a-b060-111e9e3593b4-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk\" (UID: \"328a0ef9-0b11-402a-b060-111e9e3593b4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk"
Sep 11 20:07:37.925279 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.925260    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e5a047b-fa22-4bb2-8a4d-69e0d890eda1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4\" (UID: \"0e5a047b-fa22-4bb2-8a4d-69e0d890eda1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4"
Sep 11 20:07:37.925342 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.925308    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e5a047b-fa22-4bb2-8a4d-69e0d890eda1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4\" (UID: \"0e5a047b-fa22-4bb2-8a4d-69e0d890eda1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4"
Sep 11 20:07:37.928582 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.928565    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9"
Sep 11 20:07:37.933520 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.933488    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx"]
Sep 11 20:07:37.936172 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:37.936138    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6wlz\" (UniqueName: \"kubernetes.io/projected/0e5a047b-fa22-4bb2-8a4d-69e0d890eda1-kube-api-access-w6wlz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4\" (UID: \"0e5a047b-fa22-4bb2-8a4d-69e0d890eda1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4"
Sep 11 20:07:38.026225 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:38.026193    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/328a0ef9-0b11-402a-b060-111e9e3593b4-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk\" (UID: \"328a0ef9-0b11-402a-b060-111e9e3593b4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk"
Sep 11 20:07:38.026361 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:38.026242    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/328a0ef9-0b11-402a-b060-111e9e3593b4-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk\" (UID: \"328a0ef9-0b11-402a-b060-111e9e3593b4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk"
Sep 11 20:07:38.026361 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:38.026272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v82b2\" (UniqueName: \"kubernetes.io/projected/65ac7474-c67f-4264-9bbe-dd5226bf7f89-kube-api-access-v82b2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx\" (UID: \"65ac7474-c67f-4264-9bbe-dd5226bf7f89\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx"
Sep 11 20:07:38.026361 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:38.026311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7hg9\" (UniqueName: \"kubernetes.io/projected/328a0ef9-0b11-402a-b060-111e9e3593b4-kube-api-access-s7hg9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk\" (UID: \"328a0ef9-0b11-402a-b060-111e9e3593b4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk"
Sep 11 20:07:38.026361 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:38.026329    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65ac7474-c67f-4264-9bbe-dd5226bf7f89-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx\" (UID: \"65ac7474-c67f-4264-9bbe-dd5226bf7f89\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx"
Sep 11 20:07:38.026361 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:38.026347    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65ac7474-c67f-4264-9bbe-dd5226bf7f89-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx\" (UID: \"65ac7474-c67f-4264-9bbe-dd5226bf7f89\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx"
Sep 11 20:07:38.026637 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:38.026613    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/328a0ef9-0b11-402a-b060-111e9e3593b4-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk\" (UID: \"328a0ef9-0b11-402a-b060-111e9e3593b4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk"
Sep 11 20:07:38.026680 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:38.026643    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/328a0ef9-0b11-402a-b060-111e9e3593b4-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk\" (UID: \"328a0ef9-0b11-402a-b060-111e9e3593b4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk"
Sep 11 20:07:38.032130 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:38.032105    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4"
Sep 11 20:07:38.034938 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:38.034917    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7hg9\" (UniqueName: \"kubernetes.io/projected/328a0ef9-0b11-402a-b060-111e9e3593b4-kube-api-access-s7hg9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk\" (UID: \"328a0ef9-0b11-402a-b060-111e9e3593b4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk"
Sep 11 20:07:38.127145 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:38.127113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v82b2\" (UniqueName: \"kubernetes.io/projected/65ac7474-c67f-4264-9bbe-dd5226bf7f89-kube-api-access-v82b2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx\" (UID: \"65ac7474-c67f-4264-9bbe-dd5226bf7f89\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx"
Sep 11 20:07:38.127266 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:38.127250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65ac7474-c67f-4264-9bbe-dd5226bf7f89-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx\" (UID: \"65ac7474-c67f-4264-9bbe-dd5226bf7f89\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx"
Sep 11 20:07:38.127308 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:38.127273    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65ac7474-c67f-4264-9bbe-dd5226bf7f89-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx\" (UID: \"65ac7474-c67f-4264-9bbe-dd5226bf7f89\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx"
Sep 11 20:07:38.127617 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:38.127601    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65ac7474-c67f-4264-9bbe-dd5226bf7f89-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx\" (UID: \"65ac7474-c67f-4264-9bbe-dd5226bf7f89\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx"
Sep 11 20:07:38.127654 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:38.127626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65ac7474-c67f-4264-9bbe-dd5226bf7f89-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx\" (UID: \"65ac7474-c67f-4264-9bbe-dd5226bf7f89\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx"
Sep 11 20:07:38.130037 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:38.130024    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk"
Sep 11 20:07:38.135927 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:38.135884    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v82b2\" (UniqueName: \"kubernetes.io/projected/65ac7474-c67f-4264-9bbe-dd5226bf7f89-kube-api-access-v82b2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx\" (UID: \"65ac7474-c67f-4264-9bbe-dd5226bf7f89\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx"
Sep 11 20:07:38.203017 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:38.202991    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9"]
Sep 11 20:07:38.204685 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:07:38.204660    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575db1d6_4eb0_4999_a540_a6051be7565f.slice/crio-6b059565055b1fac60bcff379542c88c96350947de56a5830c8f893d1f853875 WatchSource:0}: Error finding container 6b059565055b1fac60bcff379542c88c96350947de56a5830c8f893d1f853875: Status 404 returned error can't find the container with id 6b059565055b1fac60bcff379542c88c96350947de56a5830c8f893d1f853875
Sep 11 20:07:38.234584 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:38.234560    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx"
Sep 11 20:07:38.293817 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:38.293783    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9" event={"ID":"575db1d6-4eb0-4999-a540-a6051be7565f","Type":"ContainerStarted","Data":"9686f832ee78e18790fa52dbd0a20006b9a54f4518feae367f4fc9212b5666e4"}
Sep 11 20:07:38.293922 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:38.293845    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9" event={"ID":"575db1d6-4eb0-4999-a540-a6051be7565f","Type":"ContainerStarted","Data":"6b059565055b1fac60bcff379542c88c96350947de56a5830c8f893d1f853875"}
Sep 11 20:07:38.309334 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:38.309313    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4"]
Sep 11 20:07:38.347322 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:07:38.347291    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e5a047b_fa22_4bb2_8a4d_69e0d890eda1.slice/crio-bebc9d42560c5db8c0565ff9ca32f24e6d894915a8747643e65ef1a3aec09ef0 WatchSource:0}: Error finding container bebc9d42560c5db8c0565ff9ca32f24e6d894915a8747643e65ef1a3aec09ef0: Status 404 returned error can't find the container with id bebc9d42560c5db8c0565ff9ca32f24e6d894915a8747643e65ef1a3aec09ef0
Sep 11 20:07:38.404543 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:38.404415    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk"]
Sep 11 20:07:38.406214 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:07:38.406193    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod328a0ef9_0b11_402a_b060_111e9e3593b4.slice/crio-46dd85989d4a6f8f254bffff14d93cd1053e2d1d8e528e6431fc16406907e83e WatchSource:0}: Error finding container 46dd85989d4a6f8f254bffff14d93cd1053e2d1d8e528e6431fc16406907e83e: Status 404 returned error can't find the container with id 46dd85989d4a6f8f254bffff14d93cd1053e2d1d8e528e6431fc16406907e83e
Sep 11 20:07:38.507615 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:38.507588    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx"]
Sep 11 20:07:38.512221 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:07:38.512196    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod328a0ef9_0b11_402a_b060_111e9e3593b4.slice/crio-conmon-85abda0d5db07c97206250548fd6961c7f1ca6f4f1fe7959db47dab7be3dce85.scope\": RecentStats: unable to find data in memory cache]"
Sep 11 20:07:38.550272 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:07:38.550250    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ac7474_c67f_4264_9bbe_dd5226bf7f89.slice/crio-4e798f657eb0b015a08070300164a3b6f112d6173f21cc802e0e8b9facb4714b WatchSource:0}: Error finding container 4e798f657eb0b015a08070300164a3b6f112d6173f21cc802e0e8b9facb4714b: Status 404 returned error can't find the container with id 4e798f657eb0b015a08070300164a3b6f112d6173f21cc802e0e8b9facb4714b
Sep 11 20:07:39.299758 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:39.299719    2580 generic.go:356] "Generic (PLEG): container finished" podID="65ac7474-c67f-4264-9bbe-dd5226bf7f89" containerID="0d980c845cdedf25f00fb2a0527cfe265734b6ddaff9c585a784edbf27be3c98" exitCode=0
Sep 11 20:07:39.300354 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:39.299808    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx" event={"ID":"65ac7474-c67f-4264-9bbe-dd5226bf7f89","Type":"ContainerDied","Data":"0d980c845cdedf25f00fb2a0527cfe265734b6ddaff9c585a784edbf27be3c98"}
Sep 11 20:07:39.300354 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:39.299868    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx" event={"ID":"65ac7474-c67f-4264-9bbe-dd5226bf7f89","Type":"ContainerStarted","Data":"4e798f657eb0b015a08070300164a3b6f112d6173f21cc802e0e8b9facb4714b"}
Sep 11 20:07:39.301426 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:39.301402    2580 generic.go:356] "Generic (PLEG): container finished" podID="575db1d6-4eb0-4999-a540-a6051be7565f" containerID="9686f832ee78e18790fa52dbd0a20006b9a54f4518feae367f4fc9212b5666e4" exitCode=0
Sep 11 20:07:39.301533 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:39.301434    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9" event={"ID":"575db1d6-4eb0-4999-a540-a6051be7565f","Type":"ContainerDied","Data":"9686f832ee78e18790fa52dbd0a20006b9a54f4518feae367f4fc9212b5666e4"}
Sep 11 20:07:39.303096 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:39.303069    2580 generic.go:356] "Generic (PLEG): container finished" podID="328a0ef9-0b11-402a-b060-111e9e3593b4" containerID="85abda0d5db07c97206250548fd6961c7f1ca6f4f1fe7959db47dab7be3dce85" exitCode=0
Sep 11 20:07:39.303201 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:39.303171    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk" event={"ID":"328a0ef9-0b11-402a-b060-111e9e3593b4","Type":"ContainerDied","Data":"85abda0d5db07c97206250548fd6961c7f1ca6f4f1fe7959db47dab7be3dce85"}
Sep 11 20:07:39.303267 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:39.303216    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk" event={"ID":"328a0ef9-0b11-402a-b060-111e9e3593b4","Type":"ContainerStarted","Data":"46dd85989d4a6f8f254bffff14d93cd1053e2d1d8e528e6431fc16406907e83e"}
Sep 11 20:07:39.304969 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:39.304952    2580 generic.go:356] "Generic (PLEG): container finished" podID="0e5a047b-fa22-4bb2-8a4d-69e0d890eda1" containerID="dd1aaf8f7fea87770698bcd0192624292862cd88ad5f58a068a3e0a018167e19" exitCode=0
Sep 11 20:07:39.305050 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:39.304989    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4" event={"ID":"0e5a047b-fa22-4bb2-8a4d-69e0d890eda1","Type":"ContainerDied","Data":"dd1aaf8f7fea87770698bcd0192624292862cd88ad5f58a068a3e0a018167e19"}
Sep 11 20:07:39.305050 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:39.305030    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4" event={"ID":"0e5a047b-fa22-4bb2-8a4d-69e0d890eda1","Type":"ContainerStarted","Data":"bebc9d42560c5db8c0565ff9ca32f24e6d894915a8747643e65ef1a3aec09ef0"}
Sep 11 20:07:41.321511 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:41.321462    2580 generic.go:356] "Generic (PLEG): container finished" podID="0e5a047b-fa22-4bb2-8a4d-69e0d890eda1" containerID="6813adade290992e50ad448c76e915771cd70d354bab79dd3aefb0dd85284cbe" exitCode=0
Sep 11 20:07:41.321936 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:41.321531    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4" event={"ID":"0e5a047b-fa22-4bb2-8a4d-69e0d890eda1","Type":"ContainerDied","Data":"6813adade290992e50ad448c76e915771cd70d354bab79dd3aefb0dd85284cbe"}
Sep 11 20:07:41.323549 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:41.323528    2580 generic.go:356] "Generic (PLEG): container finished" podID="65ac7474-c67f-4264-9bbe-dd5226bf7f89" containerID="319d5fbda584f46dc6a5f9385b90e616df68f2d0ef6005bca6215a521b6891de" exitCode=0
Sep 11 20:07:41.323645 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:41.323613    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx" event={"ID":"65ac7474-c67f-4264-9bbe-dd5226bf7f89","Type":"ContainerDied","Data":"319d5fbda584f46dc6a5f9385b90e616df68f2d0ef6005bca6215a521b6891de"}
Sep 11 20:07:41.325628 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:41.325601    2580 generic.go:356] "Generic (PLEG): container finished" podID="328a0ef9-0b11-402a-b060-111e9e3593b4" containerID="c99e10928cbe3038e95296d377d40841f788cec501ee7975ff5b5c56edaf893a" exitCode=0
Sep 11 20:07:41.325710 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:41.325636    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk" event={"ID":"328a0ef9-0b11-402a-b060-111e9e3593b4","Type":"ContainerDied","Data":"c99e10928cbe3038e95296d377d40841f788cec501ee7975ff5b5c56edaf893a"}
Sep 11 20:07:42.332145 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:42.332109    2580 generic.go:356] "Generic (PLEG): container finished" podID="0e5a047b-fa22-4bb2-8a4d-69e0d890eda1" containerID="84a79c591c36eb8949b25fc8c591e4331f307801614300e1bbce5a6962585c85" exitCode=0
Sep 11 20:07:42.332545 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:42.332186    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4" event={"ID":"0e5a047b-fa22-4bb2-8a4d-69e0d890eda1","Type":"ContainerDied","Data":"84a79c591c36eb8949b25fc8c591e4331f307801614300e1bbce5a6962585c85"}
Sep 11 20:07:42.334229 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:42.334210    2580 generic.go:356] "Generic (PLEG): container finished" podID="65ac7474-c67f-4264-9bbe-dd5226bf7f89" containerID="6352118c69abe1d05660030ee012770b2afbdb15966db3361ed87d0f6c5b10ec" exitCode=0
Sep 11 20:07:42.334307 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:42.334288    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx" event={"ID":"65ac7474-c67f-4264-9bbe-dd5226bf7f89","Type":"ContainerDied","Data":"6352118c69abe1d05660030ee012770b2afbdb15966db3361ed87d0f6c5b10ec"}
Sep 11 20:07:42.336222 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:42.336197    2580 generic.go:356] "Generic (PLEG): container finished" podID="328a0ef9-0b11-402a-b060-111e9e3593b4" containerID="7331387c70fe0e7362dc8ba7564234b91ca3c60ea77c36433ee3e50ea2016763" exitCode=0
Sep 11 20:07:42.336335 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:42.336224    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk" event={"ID":"328a0ef9-0b11-402a-b060-111e9e3593b4","Type":"ContainerDied","Data":"7331387c70fe0e7362dc8ba7564234b91ca3c60ea77c36433ee3e50ea2016763"}
Sep 11 20:07:43.469447 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.469421    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4"
Sep 11 20:07:43.505197 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.505171    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk"
Sep 11 20:07:43.508965 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.508946    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx"
Sep 11 20:07:43.570273 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.570247    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v82b2\" (UniqueName: \"kubernetes.io/projected/65ac7474-c67f-4264-9bbe-dd5226bf7f89-kube-api-access-v82b2\") pod \"65ac7474-c67f-4264-9bbe-dd5226bf7f89\" (UID: \"65ac7474-c67f-4264-9bbe-dd5226bf7f89\") "
Sep 11 20:07:43.570420 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.570284    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/328a0ef9-0b11-402a-b060-111e9e3593b4-bundle\") pod \"328a0ef9-0b11-402a-b060-111e9e3593b4\" (UID: \"328a0ef9-0b11-402a-b060-111e9e3593b4\") "
Sep 11 20:07:43.570420 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.570310    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/328a0ef9-0b11-402a-b060-111e9e3593b4-util\") pod \"328a0ef9-0b11-402a-b060-111e9e3593b4\" (UID: \"328a0ef9-0b11-402a-b060-111e9e3593b4\") "
Sep 11 20:07:43.570420 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.570335    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e5a047b-fa22-4bb2-8a4d-69e0d890eda1-bundle\") pod \"0e5a047b-fa22-4bb2-8a4d-69e0d890eda1\" (UID: \"0e5a047b-fa22-4bb2-8a4d-69e0d890eda1\") "
Sep 11 20:07:43.570420 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.570361    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e5a047b-fa22-4bb2-8a4d-69e0d890eda1-util\") pod \"0e5a047b-fa22-4bb2-8a4d-69e0d890eda1\" (UID: \"0e5a047b-fa22-4bb2-8a4d-69e0d890eda1\") "
Sep 11 20:07:43.570420 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.570391    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65ac7474-c67f-4264-9bbe-dd5226bf7f89-util\") pod \"65ac7474-c67f-4264-9bbe-dd5226bf7f89\" (UID: \"65ac7474-c67f-4264-9bbe-dd5226bf7f89\") "
Sep 11 20:07:43.570721 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.570425    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6wlz\" (UniqueName: \"kubernetes.io/projected/0e5a047b-fa22-4bb2-8a4d-69e0d890eda1-kube-api-access-w6wlz\") pod \"0e5a047b-fa22-4bb2-8a4d-69e0d890eda1\" (UID: \"0e5a047b-fa22-4bb2-8a4d-69e0d890eda1\") "
Sep 11 20:07:43.570721 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.570455    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7hg9\" (UniqueName: \"kubernetes.io/projected/328a0ef9-0b11-402a-b060-111e9e3593b4-kube-api-access-s7hg9\") pod \"328a0ef9-0b11-402a-b060-111e9e3593b4\" (UID: \"328a0ef9-0b11-402a-b060-111e9e3593b4\") "
Sep 11 20:07:43.570721 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.570529    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65ac7474-c67f-4264-9bbe-dd5226bf7f89-bundle\") pod \"65ac7474-c67f-4264-9bbe-dd5226bf7f89\" (UID: \"65ac7474-c67f-4264-9bbe-dd5226bf7f89\") "
Sep 11 20:07:43.570972 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.570936    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5a047b-fa22-4bb2-8a4d-69e0d890eda1-bundle" (OuterVolumeSpecName: "bundle") pod "0e5a047b-fa22-4bb2-8a4d-69e0d890eda1" (UID: "0e5a047b-fa22-4bb2-8a4d-69e0d890eda1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:07:43.571199 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.571177    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328a0ef9-0b11-402a-b060-111e9e3593b4-bundle" (OuterVolumeSpecName: "bundle") pod "328a0ef9-0b11-402a-b060-111e9e3593b4" (UID: "328a0ef9-0b11-402a-b060-111e9e3593b4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:07:43.571648 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.571625    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ac7474-c67f-4264-9bbe-dd5226bf7f89-bundle" (OuterVolumeSpecName: "bundle") pod "65ac7474-c67f-4264-9bbe-dd5226bf7f89" (UID: "65ac7474-c67f-4264-9bbe-dd5226bf7f89"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:07:43.572568 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.572548    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ac7474-c67f-4264-9bbe-dd5226bf7f89-kube-api-access-v82b2" (OuterVolumeSpecName: "kube-api-access-v82b2") pod "65ac7474-c67f-4264-9bbe-dd5226bf7f89" (UID: "65ac7474-c67f-4264-9bbe-dd5226bf7f89"). InnerVolumeSpecName "kube-api-access-v82b2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:07:43.573170 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.573147    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e5a047b-fa22-4bb2-8a4d-69e0d890eda1-kube-api-access-w6wlz" (OuterVolumeSpecName: "kube-api-access-w6wlz") pod "0e5a047b-fa22-4bb2-8a4d-69e0d890eda1" (UID: "0e5a047b-fa22-4bb2-8a4d-69e0d890eda1"). InnerVolumeSpecName "kube-api-access-w6wlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:07:43.573170 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.573158    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/328a0ef9-0b11-402a-b060-111e9e3593b4-kube-api-access-s7hg9" (OuterVolumeSpecName: "kube-api-access-s7hg9") pod "328a0ef9-0b11-402a-b060-111e9e3593b4" (UID: "328a0ef9-0b11-402a-b060-111e9e3593b4"). InnerVolumeSpecName "kube-api-access-s7hg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:07:43.576885 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.576855    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ac7474-c67f-4264-9bbe-dd5226bf7f89-util" (OuterVolumeSpecName: "util") pod "65ac7474-c67f-4264-9bbe-dd5226bf7f89" (UID: "65ac7474-c67f-4264-9bbe-dd5226bf7f89"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:07:43.671588 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.671530    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/65ac7474-c67f-4264-9bbe-dd5226bf7f89-bundle\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:07:43.671588 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.671552    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v82b2\" (UniqueName: \"kubernetes.io/projected/65ac7474-c67f-4264-9bbe-dd5226bf7f89-kube-api-access-v82b2\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:07:43.671588 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.671564    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/328a0ef9-0b11-402a-b060-111e9e3593b4-bundle\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:07:43.671588 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.671574    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e5a047b-fa22-4bb2-8a4d-69e0d890eda1-bundle\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:07:43.671588 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.671586    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/65ac7474-c67f-4264-9bbe-dd5226bf7f89-util\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:07:43.671799 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.671595    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6wlz\" (UniqueName: \"kubernetes.io/projected/0e5a047b-fa22-4bb2-8a4d-69e0d890eda1-kube-api-access-w6wlz\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:07:43.671799 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.671604    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7hg9\" (UniqueName: \"kubernetes.io/projected/328a0ef9-0b11-402a-b060-111e9e3593b4-kube-api-access-s7hg9\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:07:43.885694 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.885631    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328a0ef9-0b11-402a-b060-111e9e3593b4-util" (OuterVolumeSpecName: "util") pod "328a0ef9-0b11-402a-b060-111e9e3593b4" (UID: "328a0ef9-0b11-402a-b060-111e9e3593b4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:07:43.885850 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.885747    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5a047b-fa22-4bb2-8a4d-69e0d890eda1-util" (OuterVolumeSpecName: "util") pod "0e5a047b-fa22-4bb2-8a4d-69e0d890eda1" (UID: "0e5a047b-fa22-4bb2-8a4d-69e0d890eda1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:07:43.973761 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.973730    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/328a0ef9-0b11-402a-b060-111e9e3593b4-util\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:07:43.973761 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:43.973758    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e5a047b-fa22-4bb2-8a4d-69e0d890eda1-util\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:07:44.347109 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:44.347024    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk"
Sep 11 20:07:44.347109 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:44.347047    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctz6mk" event={"ID":"328a0ef9-0b11-402a-b060-111e9e3593b4","Type":"ContainerDied","Data":"46dd85989d4a6f8f254bffff14d93cd1053e2d1d8e528e6431fc16406907e83e"}
Sep 11 20:07:44.347109 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:44.347099    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46dd85989d4a6f8f254bffff14d93cd1053e2d1d8e528e6431fc16406907e83e"
Sep 11 20:07:44.349011 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:44.348988    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4"
Sep 11 20:07:44.349139 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:44.348992    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngrj4" event={"ID":"0e5a047b-fa22-4bb2-8a4d-69e0d890eda1","Type":"ContainerDied","Data":"bebc9d42560c5db8c0565ff9ca32f24e6d894915a8747643e65ef1a3aec09ef0"}
Sep 11 20:07:44.349139 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:44.349098    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bebc9d42560c5db8c0565ff9ca32f24e6d894915a8747643e65ef1a3aec09ef0"
Sep 11 20:07:44.351107 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:44.351086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx" event={"ID":"65ac7474-c67f-4264-9bbe-dd5226bf7f89","Type":"ContainerDied","Data":"4e798f657eb0b015a08070300164a3b6f112d6173f21cc802e0e8b9facb4714b"}
Sep 11 20:07:44.351210 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:44.351115    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kcrmx"
Sep 11 20:07:44.351210 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:44.351120    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e798f657eb0b015a08070300164a3b6f112d6173f21cc802e0e8b9facb4714b"
Sep 11 20:07:52.388962 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:52.388882    2580 generic.go:356] "Generic (PLEG): container finished" podID="575db1d6-4eb0-4999-a540-a6051be7565f" containerID="9e53110b2e530aadc897362fc443c4fda4881583f3a7c3e81e739ac1b24ed447" exitCode=0
Sep 11 20:07:52.388962 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:52.388932    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9" event={"ID":"575db1d6-4eb0-4999-a540-a6051be7565f","Type":"ContainerDied","Data":"9e53110b2e530aadc897362fc443c4fda4881583f3a7c3e81e739ac1b24ed447"}
Sep 11 20:07:53.395964 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:53.395889    2580 generic.go:356] "Generic (PLEG): container finished" podID="575db1d6-4eb0-4999-a540-a6051be7565f" containerID="38d288111d2bdf806c3b4b2cf275095858ac55a7ded2962d9c8b48d4f87cf3ce" exitCode=0
Sep 11 20:07:53.396334 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:53.395964    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9" event={"ID":"575db1d6-4eb0-4999-a540-a6051be7565f","Type":"ContainerDied","Data":"38d288111d2bdf806c3b4b2cf275095858ac55a7ded2962d9c8b48d4f87cf3ce"}
Sep 11 20:07:54.518560 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:54.518539    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9"
Sep 11 20:07:54.657384 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:54.657321    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/575db1d6-4eb0-4999-a540-a6051be7565f-util\") pod \"575db1d6-4eb0-4999-a540-a6051be7565f\" (UID: \"575db1d6-4eb0-4999-a540-a6051be7565f\") "
Sep 11 20:07:54.657384 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:54.657366    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nk6f\" (UniqueName: \"kubernetes.io/projected/575db1d6-4eb0-4999-a540-a6051be7565f-kube-api-access-4nk6f\") pod \"575db1d6-4eb0-4999-a540-a6051be7565f\" (UID: \"575db1d6-4eb0-4999-a540-a6051be7565f\") "
Sep 11 20:07:54.657603 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:54.657396    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/575db1d6-4eb0-4999-a540-a6051be7565f-bundle\") pod \"575db1d6-4eb0-4999-a540-a6051be7565f\" (UID: \"575db1d6-4eb0-4999-a540-a6051be7565f\") "
Sep 11 20:07:54.658038 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:54.658013    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575db1d6-4eb0-4999-a540-a6051be7565f-bundle" (OuterVolumeSpecName: "bundle") pod "575db1d6-4eb0-4999-a540-a6051be7565f" (UID: "575db1d6-4eb0-4999-a540-a6051be7565f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:07:54.659466 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:54.659431    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575db1d6-4eb0-4999-a540-a6051be7565f-kube-api-access-4nk6f" (OuterVolumeSpecName: "kube-api-access-4nk6f") pod "575db1d6-4eb0-4999-a540-a6051be7565f" (UID: "575db1d6-4eb0-4999-a540-a6051be7565f"). InnerVolumeSpecName "kube-api-access-4nk6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:07:54.662083 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:54.662063    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575db1d6-4eb0-4999-a540-a6051be7565f-util" (OuterVolumeSpecName: "util") pod "575db1d6-4eb0-4999-a540-a6051be7565f" (UID: "575db1d6-4eb0-4999-a540-a6051be7565f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:07:54.758227 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:54.758203    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/575db1d6-4eb0-4999-a540-a6051be7565f-bundle\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:07:54.758227 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:54.758223    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/575db1d6-4eb0-4999-a540-a6051be7565f-util\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:07:54.758347 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:54.758232    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nk6f\" (UniqueName: \"kubernetes.io/projected/575db1d6-4eb0-4999-a540-a6051be7565f-kube-api-access-4nk6f\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:07:55.407775 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:55.407748    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9"
Sep 11 20:07:55.407936 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:55.407743    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aknrv9" event={"ID":"575db1d6-4eb0-4999-a540-a6051be7565f","Type":"ContainerDied","Data":"6b059565055b1fac60bcff379542c88c96350947de56a5830c8f893d1f853875"}
Sep 11 20:07:55.407936 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:07:55.407871    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b059565055b1fac60bcff379542c88c96350947de56a5830c8f893d1f853875"
Sep 11 20:08:12.867761 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.867727    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tx4dp"]
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.867997    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e5a047b-fa22-4bb2-8a4d-69e0d890eda1" containerName="pull"
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868008    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e5a047b-fa22-4bb2-8a4d-69e0d890eda1" containerName="pull"
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868018    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="328a0ef9-0b11-402a-b060-111e9e3593b4" containerName="pull"
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868024    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="328a0ef9-0b11-402a-b060-111e9e3593b4" containerName="pull"
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868035    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="575db1d6-4eb0-4999-a540-a6051be7565f" containerName="util"
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868041    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="575db1d6-4eb0-4999-a540-a6051be7565f" containerName="util"
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868048    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="575db1d6-4eb0-4999-a540-a6051be7565f" containerName="pull"
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868053    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="575db1d6-4eb0-4999-a540-a6051be7565f" containerName="pull"
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868064    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="328a0ef9-0b11-402a-b060-111e9e3593b4" containerName="util"
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868070    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="328a0ef9-0b11-402a-b060-111e9e3593b4" containerName="util"
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868077    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65ac7474-c67f-4264-9bbe-dd5226bf7f89" containerName="extract"
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868083    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="65ac7474-c67f-4264-9bbe-dd5226bf7f89" containerName="extract"
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868088    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65ac7474-c67f-4264-9bbe-dd5226bf7f89" containerName="util"
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868093    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="65ac7474-c67f-4264-9bbe-dd5226bf7f89" containerName="util"
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868098    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65ac7474-c67f-4264-9bbe-dd5226bf7f89" containerName="pull"
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868104    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="65ac7474-c67f-4264-9bbe-dd5226bf7f89" containerName="pull"
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868109    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e5a047b-fa22-4bb2-8a4d-69e0d890eda1" containerName="extract"
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868114    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e5a047b-fa22-4bb2-8a4d-69e0d890eda1" containerName="extract"
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868119    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e5a047b-fa22-4bb2-8a4d-69e0d890eda1" containerName="util"
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868124    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e5a047b-fa22-4bb2-8a4d-69e0d890eda1" containerName="util"
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868131    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="575db1d6-4eb0-4999-a540-a6051be7565f" containerName="extract"
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868135    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="575db1d6-4eb0-4999-a540-a6051be7565f" containerName="extract"
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868141    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="328a0ef9-0b11-402a-b060-111e9e3593b4" containerName="extract"
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868146    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="328a0ef9-0b11-402a-b060-111e9e3593b4" containerName="extract"
Sep 11 20:08:12.868190 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868199    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="575db1d6-4eb0-4999-a540-a6051be7565f" containerName="extract"
Sep 11 20:08:12.868962 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868206    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="65ac7474-c67f-4264-9bbe-dd5226bf7f89" containerName="extract"
Sep 11 20:08:12.868962 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868211    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e5a047b-fa22-4bb2-8a4d-69e0d890eda1" containerName="extract"
Sep 11 20:08:12.868962 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.868216    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="328a0ef9-0b11-402a-b060-111e9e3593b4" containerName="extract"
Sep 11 20:08:12.872807 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.872787    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tx4dp"
Sep 11 20:08:12.875485 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.875465    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 11 20:08:12.875612 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.875524    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 20:08:12.875612 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.875584    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 20:08:12.876567 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.876546    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-4flj2\""
Sep 11 20:08:12.882995 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:12.882973    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tx4dp"]
Sep 11 20:08:13.004317 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:13.004289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5jhr\" (UniqueName: \"kubernetes.io/projected/2f73cdfa-3681-4c39-915b-366711653f6f-kube-api-access-d5jhr\") pod \"dns-operator-controller-manager-7b84cd7c9c-tx4dp\" (UID: \"2f73cdfa-3681-4c39-915b-366711653f6f\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tx4dp"
Sep 11 20:08:13.105535 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:13.105487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5jhr\" (UniqueName: \"kubernetes.io/projected/2f73cdfa-3681-4c39-915b-366711653f6f-kube-api-access-d5jhr\") pod \"dns-operator-controller-manager-7b84cd7c9c-tx4dp\" (UID: \"2f73cdfa-3681-4c39-915b-366711653f6f\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tx4dp"
Sep 11 20:08:13.113884 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:13.113863    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5jhr\" (UniqueName: \"kubernetes.io/projected/2f73cdfa-3681-4c39-915b-366711653f6f-kube-api-access-d5jhr\") pod \"dns-operator-controller-manager-7b84cd7c9c-tx4dp\" (UID: \"2f73cdfa-3681-4c39-915b-366711653f6f\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tx4dp"
Sep 11 20:08:13.183859 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:13.183780    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tx4dp"
Sep 11 20:08:13.238714 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:13.238597    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-5dnxt"]
Sep 11 20:08:13.242693 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:13.242672    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5dnxt"
Sep 11 20:08:13.245102 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:13.245079    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 11 20:08:13.245215 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:13.245129    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 11 20:08:13.245388 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:13.245368    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-65m5k\""
Sep 11 20:08:13.249417 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:13.249366    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-5dnxt"]
Sep 11 20:08:13.407692 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:13.407662    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6bd712e-5be3-4bee-8974-76dff2193cac-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-5dnxt\" (UID: \"e6bd712e-5be3-4bee-8974-76dff2193cac\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5dnxt"
Sep 11 20:08:13.407840 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:13.407729    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df9xv\" (UniqueName: \"kubernetes.io/projected/e6bd712e-5be3-4bee-8974-76dff2193cac-kube-api-access-df9xv\") pod \"kuadrant-console-plugin-fbdd4f445-5dnxt\" (UID: \"e6bd712e-5be3-4bee-8974-76dff2193cac\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5dnxt"
Sep 11 20:08:13.407840 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:13.407754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e6bd712e-5be3-4bee-8974-76dff2193cac-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-5dnxt\" (UID: \"e6bd712e-5be3-4bee-8974-76dff2193cac\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5dnxt"
Sep 11 20:08:13.462513 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:13.462476    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tx4dp"]
Sep 11 20:08:13.464869 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:08:13.464845    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f73cdfa_3681_4c39_915b_366711653f6f.slice/crio-690a9b5741bc97973f702a4b6d5695f1a744d41d5471b0e49c47cea7d67f8982 WatchSource:0}: Error finding container 690a9b5741bc97973f702a4b6d5695f1a744d41d5471b0e49c47cea7d67f8982: Status 404 returned error can't find the container with id 690a9b5741bc97973f702a4b6d5695f1a744d41d5471b0e49c47cea7d67f8982
Sep 11 20:08:13.490287 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:13.490262    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tx4dp" event={"ID":"2f73cdfa-3681-4c39-915b-366711653f6f","Type":"ContainerStarted","Data":"690a9b5741bc97973f702a4b6d5695f1a744d41d5471b0e49c47cea7d67f8982"}
Sep 11 20:08:13.508587 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:13.508564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9xv\" (UniqueName: \"kubernetes.io/projected/e6bd712e-5be3-4bee-8974-76dff2193cac-kube-api-access-df9xv\") pod \"kuadrant-console-plugin-fbdd4f445-5dnxt\" (UID: \"e6bd712e-5be3-4bee-8974-76dff2193cac\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5dnxt"
Sep 11 20:08:13.508677 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:13.508595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e6bd712e-5be3-4bee-8974-76dff2193cac-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-5dnxt\" (UID: \"e6bd712e-5be3-4bee-8974-76dff2193cac\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5dnxt"
Sep 11 20:08:13.508677 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:13.508631    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6bd712e-5be3-4bee-8974-76dff2193cac-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-5dnxt\" (UID: \"e6bd712e-5be3-4bee-8974-76dff2193cac\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5dnxt"
Sep 11 20:08:13.508760 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:08:13.508735    2580 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 11 20:08:13.508803 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:08:13.508788    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6bd712e-5be3-4bee-8974-76dff2193cac-plugin-serving-cert podName:e6bd712e-5be3-4bee-8974-76dff2193cac nodeName:}" failed. No retries permitted until 2026-09-11 20:08:14.008770159 +0000 UTC m=+464.000932424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/e6bd712e-5be3-4bee-8974-76dff2193cac-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-5dnxt" (UID: "e6bd712e-5be3-4bee-8974-76dff2193cac") : secret "plugin-serving-cert" not found
Sep 11 20:08:13.509175 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:13.509156    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e6bd712e-5be3-4bee-8974-76dff2193cac-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-5dnxt\" (UID: \"e6bd712e-5be3-4bee-8974-76dff2193cac\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5dnxt"
Sep 11 20:08:13.520246 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:13.520222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df9xv\" (UniqueName: \"kubernetes.io/projected/e6bd712e-5be3-4bee-8974-76dff2193cac-kube-api-access-df9xv\") pod \"kuadrant-console-plugin-fbdd4f445-5dnxt\" (UID: \"e6bd712e-5be3-4bee-8974-76dff2193cac\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5dnxt"
Sep 11 20:08:14.014009 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:14.013930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6bd712e-5be3-4bee-8974-76dff2193cac-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-5dnxt\" (UID: \"e6bd712e-5be3-4bee-8974-76dff2193cac\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5dnxt"
Sep 11 20:08:14.017072 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:14.017037    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6bd712e-5be3-4bee-8974-76dff2193cac-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-5dnxt\" (UID: \"e6bd712e-5be3-4bee-8974-76dff2193cac\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5dnxt"
Sep 11 20:08:14.157098 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:14.157062    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5dnxt"
Sep 11 20:08:14.432978 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:14.432915    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-5dnxt"]
Sep 11 20:08:14.434579 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:08:14.434545    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6bd712e_5be3_4bee_8974_76dff2193cac.slice/crio-a20e7587451c251c7cc98764eb4c0a1b3463604860408f323ca6d0697c0291d0 WatchSource:0}: Error finding container a20e7587451c251c7cc98764eb4c0a1b3463604860408f323ca6d0697c0291d0: Status 404 returned error can't find the container with id a20e7587451c251c7cc98764eb4c0a1b3463604860408f323ca6d0697c0291d0
Sep 11 20:08:14.496850 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:14.496822    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5dnxt" event={"ID":"e6bd712e-5be3-4bee-8974-76dff2193cac","Type":"ContainerStarted","Data":"a20e7587451c251c7cc98764eb4c0a1b3463604860408f323ca6d0697c0291d0"}
Sep 11 20:08:16.512283 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:16.512211    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tx4dp" event={"ID":"2f73cdfa-3681-4c39-915b-366711653f6f","Type":"ContainerStarted","Data":"d72a63afdc6a0b2a5ff9c65f072d1411c0051d281074ea31ff33b69de89cff13"}
Sep 11 20:08:16.512878 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:16.512819    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tx4dp"
Sep 11 20:08:16.532931 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:16.532889    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tx4dp" podStartSLOduration=1.9267349459999998 podStartE2EDuration="4.532875077s" totalImagesPullingTime="2.606140131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:08:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:08:16.530808433 +0000 UTC m=+466.522970705" watchObservedRunningTime="2026-09-11 20:08:16.532875077 +0000 UTC m=+466.525037347"
Sep 11 20:08:20.534478 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:20.534440    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5dnxt" event={"ID":"e6bd712e-5be3-4bee-8974-76dff2193cac","Type":"ContainerStarted","Data":"68935945b36789ef610bd48c05d572483e9362c3b0717cb86000d460f469ae6e"}
Sep 11 20:08:20.550740 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:20.550693    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5dnxt" podStartSLOduration=2.121873341 podStartE2EDuration="7.550673433s" totalImagesPullingTime="5.428800092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:08:20.548949115 +0000 UTC m=+470.541111386" watchObservedRunningTime="2026-09-11 20:08:20.550673433 +0000 UTC m=+470.542835704"
Sep 11 20:08:28.524577 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:08:28.524545    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tx4dp"
Sep 11 20:09:00.175889 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.175851    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:09:00.178914 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.178898    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"
Sep 11 20:09:00.181477 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.181457    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 20:09:00.188574 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.188550    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:09:00.216582 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.216555    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:09:00.239272 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.239251    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqgs6\" (UniqueName: \"kubernetes.io/projected/6f53db6b-18c9-46a8-801a-da9c44c8bc17-kube-api-access-sqgs6\") pod \"limitador-limitador-84994f7ffc-7kjmw\" (UID: \"6f53db6b-18c9-46a8-801a-da9c44c8bc17\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"
Sep 11 20:09:00.239342 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.239286    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6f53db6b-18c9-46a8-801a-da9c44c8bc17-config-file\") pod \"limitador-limitador-84994f7ffc-7kjmw\" (UID: \"6f53db6b-18c9-46a8-801a-da9c44c8bc17\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"
Sep 11 20:09:00.340671 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.340636    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqgs6\" (UniqueName: \"kubernetes.io/projected/6f53db6b-18c9-46a8-801a-da9c44c8bc17-kube-api-access-sqgs6\") pod \"limitador-limitador-84994f7ffc-7kjmw\" (UID: \"6f53db6b-18c9-46a8-801a-da9c44c8bc17\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"
Sep 11 20:09:00.340802 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.340687    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6f53db6b-18c9-46a8-801a-da9c44c8bc17-config-file\") pod \"limitador-limitador-84994f7ffc-7kjmw\" (UID: \"6f53db6b-18c9-46a8-801a-da9c44c8bc17\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"
Sep 11 20:09:00.341250 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.341232    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6f53db6b-18c9-46a8-801a-da9c44c8bc17-config-file\") pod \"limitador-limitador-84994f7ffc-7kjmw\" (UID: \"6f53db6b-18c9-46a8-801a-da9c44c8bc17\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"
Sep 11 20:09:00.348454 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.348425    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqgs6\" (UniqueName: \"kubernetes.io/projected/6f53db6b-18c9-46a8-801a-da9c44c8bc17-kube-api-access-sqgs6\") pod \"limitador-limitador-84994f7ffc-7kjmw\" (UID: \"6f53db6b-18c9-46a8-801a-da9c44c8bc17\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"
Sep 11 20:09:00.489816 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.489785    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"
Sep 11 20:09:00.609815 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.609778    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-pl6sx"]
Sep 11 20:09:00.613120 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.613103    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-pl6sx"
Sep 11 20:09:00.616383 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.616363    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5tkpk\""
Sep 11 20:09:00.621894 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.621872    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-pl6sx"]
Sep 11 20:09:00.642650 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.642627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl5pw\" (UniqueName: \"kubernetes.io/projected/104a1beb-2371-4086-8c19-0a663d3f9cb9-kube-api-access-fl5pw\") pod \"authorino-7bcfd8f9cd-pl6sx\" (UID: \"104a1beb-2371-4086-8c19-0a663d3f9cb9\") " pod="kuadrant-system/authorino-7bcfd8f9cd-pl6sx"
Sep 11 20:09:00.721747 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.721716    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-lpdbs"]
Sep 11 20:09:00.724861 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.724846    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-lpdbs"
Sep 11 20:09:00.732241 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.732218    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lpdbs"]
Sep 11 20:09:00.743630 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.743565    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl5pw\" (UniqueName: \"kubernetes.io/projected/104a1beb-2371-4086-8c19-0a663d3f9cb9-kube-api-access-fl5pw\") pod \"authorino-7bcfd8f9cd-pl6sx\" (UID: \"104a1beb-2371-4086-8c19-0a663d3f9cb9\") " pod="kuadrant-system/authorino-7bcfd8f9cd-pl6sx"
Sep 11 20:09:00.743736 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.743674    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clkhq\" (UniqueName: \"kubernetes.io/projected/1c26916b-508b-4ad3-97de-f53e62577fbb-kube-api-access-clkhq\") pod \"authorino-59c974bd48-lpdbs\" (UID: \"1c26916b-508b-4ad3-97de-f53e62577fbb\") " pod="kuadrant-system/authorino-59c974bd48-lpdbs"
Sep 11 20:09:00.752014 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.751996    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl5pw\" (UniqueName: \"kubernetes.io/projected/104a1beb-2371-4086-8c19-0a663d3f9cb9-kube-api-access-fl5pw\") pod \"authorino-7bcfd8f9cd-pl6sx\" (UID: \"104a1beb-2371-4086-8c19-0a663d3f9cb9\") " pod="kuadrant-system/authorino-7bcfd8f9cd-pl6sx"
Sep 11 20:09:00.760135 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.760042    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:09:00.762173 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:09:00.762153    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f53db6b_18c9_46a8_801a_da9c44c8bc17.slice/crio-d4eec2269b9bcfc818697d5f2f7382282b5b8ca30544e88a0226cd5ddcbfded6 WatchSource:0}: Error finding container d4eec2269b9bcfc818697d5f2f7382282b5b8ca30544e88a0226cd5ddcbfded6: Status 404 returned error can't find the container with id d4eec2269b9bcfc818697d5f2f7382282b5b8ca30544e88a0226cd5ddcbfded6
Sep 11 20:09:00.844417 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.844394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clkhq\" (UniqueName: \"kubernetes.io/projected/1c26916b-508b-4ad3-97de-f53e62577fbb-kube-api-access-clkhq\") pod \"authorino-59c974bd48-lpdbs\" (UID: \"1c26916b-508b-4ad3-97de-f53e62577fbb\") " pod="kuadrant-system/authorino-59c974bd48-lpdbs"
Sep 11 20:09:00.852785 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.852766    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clkhq\" (UniqueName: \"kubernetes.io/projected/1c26916b-508b-4ad3-97de-f53e62577fbb-kube-api-access-clkhq\") pod \"authorino-59c974bd48-lpdbs\" (UID: \"1c26916b-508b-4ad3-97de-f53e62577fbb\") " pod="kuadrant-system/authorino-59c974bd48-lpdbs"
Sep 11 20:09:00.923161 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:00.923142    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-pl6sx"
Sep 11 20:09:01.034420 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:01.034354    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-lpdbs"
Sep 11 20:09:01.190981 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:01.190956    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-pl6sx"]
Sep 11 20:09:01.192449 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:09:01.192425    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod104a1beb_2371_4086_8c19_0a663d3f9cb9.slice/crio-081e637684281624dd87ed0758ec9ea120e36becf50183deef04ea25a30997e5 WatchSource:0}: Error finding container 081e637684281624dd87ed0758ec9ea120e36becf50183deef04ea25a30997e5: Status 404 returned error can't find the container with id 081e637684281624dd87ed0758ec9ea120e36becf50183deef04ea25a30997e5
Sep 11 20:09:01.300969 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:01.300946    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lpdbs"]
Sep 11 20:09:01.302635 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:09:01.302608    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c26916b_508b_4ad3_97de_f53e62577fbb.slice/crio-c75ccbf105d12df70187501f71c5fef451848341834d144b564a92d6df1fc596 WatchSource:0}: Error finding container c75ccbf105d12df70187501f71c5fef451848341834d144b564a92d6df1fc596: Status 404 returned error can't find the container with id c75ccbf105d12df70187501f71c5fef451848341834d144b564a92d6df1fc596
Sep 11 20:09:01.740621 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:01.740583    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-pl6sx" event={"ID":"104a1beb-2371-4086-8c19-0a663d3f9cb9","Type":"ContainerStarted","Data":"081e637684281624dd87ed0758ec9ea120e36becf50183deef04ea25a30997e5"}
Sep 11 20:09:01.741785 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:01.741759    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-7kjmw" event={"ID":"6f53db6b-18c9-46a8-801a-da9c44c8bc17","Type":"ContainerStarted","Data":"d4eec2269b9bcfc818697d5f2f7382282b5b8ca30544e88a0226cd5ddcbfded6"}
Sep 11 20:09:01.742904 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:01.742879    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-lpdbs" event={"ID":"1c26916b-508b-4ad3-97de-f53e62577fbb","Type":"ContainerStarted","Data":"c75ccbf105d12df70187501f71c5fef451848341834d144b564a92d6df1fc596"}
Sep 11 20:09:02.749840 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:02.749802    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-7kjmw" event={"ID":"6f53db6b-18c9-46a8-801a-da9c44c8bc17","Type":"ContainerStarted","Data":"2e58dbe76fc016021ffe688af594e5774e37acf910e9ffba8314adf0513f03e5"}
Sep 11 20:09:02.750247 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:02.749959    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"
Sep 11 20:09:02.769623 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:02.769570    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-7kjmw" podStartSLOduration=0.916486276 podStartE2EDuration="2.769554557s" totalImagesPullingTime="1.853068281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:09:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:09:02.767781914 +0000 UTC m=+512.759944187" watchObservedRunningTime="2026-09-11 20:09:02.769554557 +0000 UTC m=+512.761716826"
Sep 11 20:09:03.755945 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:03.755904    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-lpdbs" event={"ID":"1c26916b-508b-4ad3-97de-f53e62577fbb","Type":"ContainerStarted","Data":"da030743b8fa947c770e38982187e68d9243c7b7af288bbd14e0aa6fb70b4649"}
Sep 11 20:09:03.770937 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:03.770890    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-lpdbs" podStartSLOduration=1.900910075 podStartE2EDuration="3.770875597s" totalImagesPullingTime="1.869965522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:09:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:09:03.769432369 +0000 UTC m=+513.761594640" watchObservedRunningTime="2026-09-11 20:09:03.770875597 +0000 UTC m=+513.763037867"
Sep 11 20:09:03.796938 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:03.796906    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-pl6sx"]
Sep 11 20:09:13.756938 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:13.756912    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"
Sep 11 20:09:13.806312 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:13.806271    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-pl6sx" event={"ID":"104a1beb-2371-4086-8c19-0a663d3f9cb9","Type":"ContainerStarted","Data":"f24bfb49c6eeb99a2aac555d7eaa9d6fab52fca18810711c79e9197b9dc689c5"}
Sep 11 20:09:13.806518 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:13.806329    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-pl6sx" podUID="104a1beb-2371-4086-8c19-0a663d3f9cb9" containerName="authorino" containerID="cri-o://f24bfb49c6eeb99a2aac555d7eaa9d6fab52fca18810711c79e9197b9dc689c5" gracePeriod=30
Sep 11 20:09:13.829023 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:13.828971    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-pl6sx" podStartSLOduration=1.291354265 podStartE2EDuration="13.828953587s" totalImagesPullingTime="12.537599322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:09:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:09:13.82694712 +0000 UTC m=+523.819109417" watchObservedRunningTime="2026-09-11 20:09:13.828953587 +0000 UTC m=+523.821115857"
Sep 11 20:09:14.054892 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:14.054868    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-pl6sx"
Sep 11 20:09:14.154852 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:14.154816    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fl5pw\" (UniqueName: \"kubernetes.io/projected/104a1beb-2371-4086-8c19-0a663d3f9cb9-kube-api-access-fl5pw\") pod \"104a1beb-2371-4086-8c19-0a663d3f9cb9\" (UID: \"104a1beb-2371-4086-8c19-0a663d3f9cb9\") "
Sep 11 20:09:14.157010 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:14.156988    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104a1beb-2371-4086-8c19-0a663d3f9cb9-kube-api-access-fl5pw" (OuterVolumeSpecName: "kube-api-access-fl5pw") pod "104a1beb-2371-4086-8c19-0a663d3f9cb9" (UID: "104a1beb-2371-4086-8c19-0a663d3f9cb9"). InnerVolumeSpecName "kube-api-access-fl5pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:09:14.255558 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:14.255521    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl5pw\" (UniqueName: \"kubernetes.io/projected/104a1beb-2371-4086-8c19-0a663d3f9cb9-kube-api-access-fl5pw\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:09:14.812955 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:14.812868    2580 generic.go:356] "Generic (PLEG): container finished" podID="104a1beb-2371-4086-8c19-0a663d3f9cb9" containerID="f24bfb49c6eeb99a2aac555d7eaa9d6fab52fca18810711c79e9197b9dc689c5" exitCode=2
Sep 11 20:09:14.812955 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:14.812924    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-pl6sx"
Sep 11 20:09:14.812955 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:14.812934    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-pl6sx" event={"ID":"104a1beb-2371-4086-8c19-0a663d3f9cb9","Type":"ContainerDied","Data":"f24bfb49c6eeb99a2aac555d7eaa9d6fab52fca18810711c79e9197b9dc689c5"}
Sep 11 20:09:14.813434 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:14.812969    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-pl6sx" event={"ID":"104a1beb-2371-4086-8c19-0a663d3f9cb9","Type":"ContainerDied","Data":"081e637684281624dd87ed0758ec9ea120e36becf50183deef04ea25a30997e5"}
Sep 11 20:09:14.813434 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:14.812987    2580 scope.go:117] "RemoveContainer" containerID="f24bfb49c6eeb99a2aac555d7eaa9d6fab52fca18810711c79e9197b9dc689c5"
Sep 11 20:09:14.821478 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:14.821457    2580 scope.go:117] "RemoveContainer" containerID="f24bfb49c6eeb99a2aac555d7eaa9d6fab52fca18810711c79e9197b9dc689c5"
Sep 11 20:09:14.821765 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:09:14.821748    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f24bfb49c6eeb99a2aac555d7eaa9d6fab52fca18810711c79e9197b9dc689c5\": container with ID starting with f24bfb49c6eeb99a2aac555d7eaa9d6fab52fca18810711c79e9197b9dc689c5 not found: ID does not exist" containerID="f24bfb49c6eeb99a2aac555d7eaa9d6fab52fca18810711c79e9197b9dc689c5"
Sep 11 20:09:14.821833 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:14.821772    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f24bfb49c6eeb99a2aac555d7eaa9d6fab52fca18810711c79e9197b9dc689c5"} err="failed to get container status \"f24bfb49c6eeb99a2aac555d7eaa9d6fab52fca18810711c79e9197b9dc689c5\": rpc error: code = NotFound desc = could not find container \"f24bfb49c6eeb99a2aac555d7eaa9d6fab52fca18810711c79e9197b9dc689c5\": container with ID starting with f24bfb49c6eeb99a2aac555d7eaa9d6fab52fca18810711c79e9197b9dc689c5 not found: ID does not exist"
Sep 11 20:09:14.834283 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:14.834261    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-pl6sx"]
Sep 11 20:09:14.838895 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:14.838871    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-pl6sx"]
Sep 11 20:09:16.440333 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:16.440302    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="104a1beb-2371-4086-8c19-0a663d3f9cb9" path="/var/lib/kubelet/pods/104a1beb-2371-4086-8c19-0a663d3f9cb9/volumes"
Sep 11 20:09:28.475926 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.475895    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-2mczb"]
Sep 11 20:09:28.476277 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.476200    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104a1beb-2371-4086-8c19-0a663d3f9cb9" containerName="authorino"
Sep 11 20:09:28.476277 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.476212    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="104a1beb-2371-4086-8c19-0a663d3f9cb9" containerName="authorino"
Sep 11 20:09:28.476277 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.476264    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="104a1beb-2371-4086-8c19-0a663d3f9cb9" containerName="authorino"
Sep 11 20:09:28.478934 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.478918    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-2mczb"
Sep 11 20:09:28.489286 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.489262    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-2mczb"]
Sep 11 20:09:28.571891 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.571850    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bh75\" (UniqueName: \"kubernetes.io/projected/52c4c281-1640-4e5d-8d97-f648405f4469-kube-api-access-9bh75\") pod \"authorino-557f876f69-2mczb\" (UID: \"52c4c281-1640-4e5d-8d97-f648405f4469\") " pod="kuadrant-system/authorino-557f876f69-2mczb"
Sep 11 20:09:28.574082 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.574058    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-2mczb"]
Sep 11 20:09:28.574263 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:09:28.574244    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-9bh75], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-2mczb" podUID="52c4c281-1640-4e5d-8d97-f648405f4469"
Sep 11 20:09:28.607582 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.607563    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-glvdr"]
Sep 11 20:09:28.610578 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.610564    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-glvdr"
Sep 11 20:09:28.613517 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.613485    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 20:09:28.619601 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.619579    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-glvdr"]
Sep 11 20:09:28.672999 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.672968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bh75\" (UniqueName: \"kubernetes.io/projected/52c4c281-1640-4e5d-8d97-f648405f4469-kube-api-access-9bh75\") pod \"authorino-557f876f69-2mczb\" (UID: \"52c4c281-1640-4e5d-8d97-f648405f4469\") " pod="kuadrant-system/authorino-557f876f69-2mczb"
Sep 11 20:09:28.687328 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.687305    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bh75\" (UniqueName: \"kubernetes.io/projected/52c4c281-1640-4e5d-8d97-f648405f4469-kube-api-access-9bh75\") pod \"authorino-557f876f69-2mczb\" (UID: \"52c4c281-1640-4e5d-8d97-f648405f4469\") " pod="kuadrant-system/authorino-557f876f69-2mczb"
Sep 11 20:09:28.690581 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.690560    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-glvdr"]
Sep 11 20:09:28.690765 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:09:28.690748    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-twkj9 tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-twkj9 tls-cert]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-glvdr" podUID="e78c3349-e7e4-43e1-8633-2be5a2da0c3f"
Sep 11 20:09:28.773539 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.773475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e78c3349-e7e4-43e1-8633-2be5a2da0c3f-tls-cert\") pod \"authorino-6d4dfc649f-glvdr\" (UID: \"e78c3349-e7e4-43e1-8633-2be5a2da0c3f\") " pod="kuadrant-system/authorino-6d4dfc649f-glvdr"
Sep 11 20:09:28.773644 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.773568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twkj9\" (UniqueName: \"kubernetes.io/projected/e78c3349-e7e4-43e1-8633-2be5a2da0c3f-kube-api-access-twkj9\") pod \"authorino-6d4dfc649f-glvdr\" (UID: \"e78c3349-e7e4-43e1-8633-2be5a2da0c3f\") " pod="kuadrant-system/authorino-6d4dfc649f-glvdr"
Sep 11 20:09:28.874075 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.874049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twkj9\" (UniqueName: \"kubernetes.io/projected/e78c3349-e7e4-43e1-8633-2be5a2da0c3f-kube-api-access-twkj9\") pod \"authorino-6d4dfc649f-glvdr\" (UID: \"e78c3349-e7e4-43e1-8633-2be5a2da0c3f\") " pod="kuadrant-system/authorino-6d4dfc649f-glvdr"
Sep 11 20:09:28.874188 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.874083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e78c3349-e7e4-43e1-8633-2be5a2da0c3f-tls-cert\") pod \"authorino-6d4dfc649f-glvdr\" (UID: \"e78c3349-e7e4-43e1-8633-2be5a2da0c3f\") " pod="kuadrant-system/authorino-6d4dfc649f-glvdr"
Sep 11 20:09:28.876839 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.876814    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e78c3349-e7e4-43e1-8633-2be5a2da0c3f-tls-cert\") pod \"authorino-6d4dfc649f-glvdr\" (UID: \"e78c3349-e7e4-43e1-8633-2be5a2da0c3f\") " pod="kuadrant-system/authorino-6d4dfc649f-glvdr"
Sep 11 20:09:28.879003 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.878982    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-glvdr"
Sep 11 20:09:28.879095 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.879004    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-2mczb"
Sep 11 20:09:28.882589 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.882569    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twkj9\" (UniqueName: \"kubernetes.io/projected/e78c3349-e7e4-43e1-8633-2be5a2da0c3f-kube-api-access-twkj9\") pod \"authorino-6d4dfc649f-glvdr\" (UID: \"e78c3349-e7e4-43e1-8633-2be5a2da0c3f\") " pod="kuadrant-system/authorino-6d4dfc649f-glvdr"
Sep 11 20:09:28.895155 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.895137    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-glvdr"
Sep 11 20:09:28.898127 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.898110    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-2mczb"
Sep 11 20:09:28.974772 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.974751    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bh75\" (UniqueName: \"kubernetes.io/projected/52c4c281-1640-4e5d-8d97-f648405f4469-kube-api-access-9bh75\") pod \"52c4c281-1640-4e5d-8d97-f648405f4469\" (UID: \"52c4c281-1640-4e5d-8d97-f648405f4469\") "
Sep 11 20:09:28.974899 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.974786    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e78c3349-e7e4-43e1-8633-2be5a2da0c3f-tls-cert\") pod \"e78c3349-e7e4-43e1-8633-2be5a2da0c3f\" (UID: \"e78c3349-e7e4-43e1-8633-2be5a2da0c3f\") "
Sep 11 20:09:28.974899 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.974822    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twkj9\" (UniqueName: \"kubernetes.io/projected/e78c3349-e7e4-43e1-8633-2be5a2da0c3f-kube-api-access-twkj9\") pod \"e78c3349-e7e4-43e1-8633-2be5a2da0c3f\" (UID: \"e78c3349-e7e4-43e1-8633-2be5a2da0c3f\") "
Sep 11 20:09:28.976897 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.976873    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e78c3349-e7e4-43e1-8633-2be5a2da0c3f-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "e78c3349-e7e4-43e1-8633-2be5a2da0c3f" (UID: "e78c3349-e7e4-43e1-8633-2be5a2da0c3f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:09:28.976897 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.976877    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c4c281-1640-4e5d-8d97-f648405f4469-kube-api-access-9bh75" (OuterVolumeSpecName: "kube-api-access-9bh75") pod "52c4c281-1640-4e5d-8d97-f648405f4469" (UID: "52c4c281-1640-4e5d-8d97-f648405f4469"). InnerVolumeSpecName "kube-api-access-9bh75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:09:28.977018 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:28.976882    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78c3349-e7e4-43e1-8633-2be5a2da0c3f-kube-api-access-twkj9" (OuterVolumeSpecName: "kube-api-access-twkj9") pod "e78c3349-e7e4-43e1-8633-2be5a2da0c3f" (UID: "e78c3349-e7e4-43e1-8633-2be5a2da0c3f"). InnerVolumeSpecName "kube-api-access-twkj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:09:29.076004 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:29.075934    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bh75\" (UniqueName: \"kubernetes.io/projected/52c4c281-1640-4e5d-8d97-f648405f4469-kube-api-access-9bh75\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:09:29.076004 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:29.075957    2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e78c3349-e7e4-43e1-8633-2be5a2da0c3f-tls-cert\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:09:29.076004 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:29.075966    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twkj9\" (UniqueName: \"kubernetes.io/projected/e78c3349-e7e4-43e1-8633-2be5a2da0c3f-kube-api-access-twkj9\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:09:29.884076 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:29.884044    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-2mczb"
Sep 11 20:09:29.884538 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:29.884049    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-glvdr"
Sep 11 20:09:29.916507 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:29.916470    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-2mczb"]
Sep 11 20:09:29.920486 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:29.920465    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-2mczb"]
Sep 11 20:09:29.941648 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:29.941628    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-glvdr"]
Sep 11 20:09:29.945472 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:29.945452    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-glvdr"]
Sep 11 20:09:30.446608 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:30.446576    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52c4c281-1640-4e5d-8d97-f648405f4469" path="/var/lib/kubelet/pods/52c4c281-1640-4e5d-8d97-f648405f4469/volumes"
Sep 11 20:09:30.446829 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:30.446817    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e78c3349-e7e4-43e1-8633-2be5a2da0c3f" path="/var/lib/kubelet/pods/e78c3349-e7e4-43e1-8633-2be5a2da0c3f/volumes"
Sep 11 20:09:31.893741 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:31.893705    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lpdbs"]
Sep 11 20:09:31.894257 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:31.893883    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-lpdbs" podUID="1c26916b-508b-4ad3-97de-f53e62577fbb" containerName="authorino" containerID="cri-o://da030743b8fa947c770e38982187e68d9243c7b7af288bbd14e0aa6fb70b4649" gracePeriod=30
Sep 11 20:09:32.137829 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:32.137805    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-lpdbs"
Sep 11 20:09:32.300586 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:32.300547    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clkhq\" (UniqueName: \"kubernetes.io/projected/1c26916b-508b-4ad3-97de-f53e62577fbb-kube-api-access-clkhq\") pod \"1c26916b-508b-4ad3-97de-f53e62577fbb\" (UID: \"1c26916b-508b-4ad3-97de-f53e62577fbb\") "
Sep 11 20:09:32.302595 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:32.302565    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c26916b-508b-4ad3-97de-f53e62577fbb-kube-api-access-clkhq" (OuterVolumeSpecName: "kube-api-access-clkhq") pod "1c26916b-508b-4ad3-97de-f53e62577fbb" (UID: "1c26916b-508b-4ad3-97de-f53e62577fbb"). InnerVolumeSpecName "kube-api-access-clkhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:09:32.401324 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:32.401295    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clkhq\" (UniqueName: \"kubernetes.io/projected/1c26916b-508b-4ad3-97de-f53e62577fbb-kube-api-access-clkhq\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:09:32.899001 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:32.898967    2580 generic.go:356] "Generic (PLEG): container finished" podID="1c26916b-508b-4ad3-97de-f53e62577fbb" containerID="da030743b8fa947c770e38982187e68d9243c7b7af288bbd14e0aa6fb70b4649" exitCode=0
Sep 11 20:09:32.899389 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:32.899008    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-lpdbs"
Sep 11 20:09:32.899389 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:32.899038    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-lpdbs" event={"ID":"1c26916b-508b-4ad3-97de-f53e62577fbb","Type":"ContainerDied","Data":"da030743b8fa947c770e38982187e68d9243c7b7af288bbd14e0aa6fb70b4649"}
Sep 11 20:09:32.899389 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:32.899067    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-lpdbs" event={"ID":"1c26916b-508b-4ad3-97de-f53e62577fbb","Type":"ContainerDied","Data":"c75ccbf105d12df70187501f71c5fef451848341834d144b564a92d6df1fc596"}
Sep 11 20:09:32.899389 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:32.899083    2580 scope.go:117] "RemoveContainer" containerID="da030743b8fa947c770e38982187e68d9243c7b7af288bbd14e0aa6fb70b4649"
Sep 11 20:09:32.907383 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:32.907363    2580 scope.go:117] "RemoveContainer" containerID="da030743b8fa947c770e38982187e68d9243c7b7af288bbd14e0aa6fb70b4649"
Sep 11 20:09:32.907656 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:09:32.907635    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da030743b8fa947c770e38982187e68d9243c7b7af288bbd14e0aa6fb70b4649\": container with ID starting with da030743b8fa947c770e38982187e68d9243c7b7af288bbd14e0aa6fb70b4649 not found: ID does not exist" containerID="da030743b8fa947c770e38982187e68d9243c7b7af288bbd14e0aa6fb70b4649"
Sep 11 20:09:32.907735 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:32.907662    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da030743b8fa947c770e38982187e68d9243c7b7af288bbd14e0aa6fb70b4649"} err="failed to get container status \"da030743b8fa947c770e38982187e68d9243c7b7af288bbd14e0aa6fb70b4649\": rpc error: code = NotFound desc = could not find container \"da030743b8fa947c770e38982187e68d9243c7b7af288bbd14e0aa6fb70b4649\": container with ID starting with da030743b8fa947c770e38982187e68d9243c7b7af288bbd14e0aa6fb70b4649 not found: ID does not exist"
Sep 11 20:09:32.939078 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:32.939054    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lpdbs"]
Sep 11 20:09:32.945812 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:32.945788    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lpdbs"]
Sep 11 20:09:34.440478 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:09:34.440444    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c26916b-508b-4ad3-97de-f53e62577fbb" path="/var/lib/kubelet/pods/1c26916b-508b-4ad3-97de-f53e62577fbb/volumes"
Sep 11 20:10:14.218510 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:14.218467    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-jtmt6"]
Sep 11 20:10:14.218953 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:14.218934    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c26916b-508b-4ad3-97de-f53e62577fbb" containerName="authorino"
Sep 11 20:10:14.219000 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:14.218959    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c26916b-508b-4ad3-97de-f53e62577fbb" containerName="authorino"
Sep 11 20:10:14.219060 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:14.219048    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c26916b-508b-4ad3-97de-f53e62577fbb" containerName="authorino"
Sep 11 20:10:14.222423 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:14.222403    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-jtmt6"
Sep 11 20:10:14.225097 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:14.225049    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-blgh8\""
Sep 11 20:10:14.225236 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:14.225214    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 20:10:14.236006 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:14.235982    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-jtmt6"]
Sep 11 20:10:14.304941 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:14.304916    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:10:14.335718 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:14.335695    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76f2ec85-7849-4143-9b55-65fe0133f662-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-jtmt6\" (UID: \"76f2ec85-7849-4143-9b55-65fe0133f662\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jtmt6"
Sep 11 20:10:14.335833 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:14.335755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h2fn\" (UniqueName: \"kubernetes.io/projected/76f2ec85-7849-4143-9b55-65fe0133f662-kube-api-access-9h2fn\") pod \"payload-processing-5b54b7fbb7-jtmt6\" (UID: \"76f2ec85-7849-4143-9b55-65fe0133f662\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jtmt6"
Sep 11 20:10:14.436535 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:14.436485    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h2fn\" (UniqueName: \"kubernetes.io/projected/76f2ec85-7849-4143-9b55-65fe0133f662-kube-api-access-9h2fn\") pod \"payload-processing-5b54b7fbb7-jtmt6\" (UID: \"76f2ec85-7849-4143-9b55-65fe0133f662\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jtmt6"
Sep 11 20:10:14.436673 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:14.436562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76f2ec85-7849-4143-9b55-65fe0133f662-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-jtmt6\" (UID: \"76f2ec85-7849-4143-9b55-65fe0133f662\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jtmt6"
Sep 11 20:10:14.437348 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:14.437325    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76f2ec85-7849-4143-9b55-65fe0133f662-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-jtmt6\" (UID: \"76f2ec85-7849-4143-9b55-65fe0133f662\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jtmt6"
Sep 11 20:10:14.445763 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:14.445735    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h2fn\" (UniqueName: \"kubernetes.io/projected/76f2ec85-7849-4143-9b55-65fe0133f662-kube-api-access-9h2fn\") pod \"payload-processing-5b54b7fbb7-jtmt6\" (UID: \"76f2ec85-7849-4143-9b55-65fe0133f662\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-jtmt6"
Sep 11 20:10:14.532663 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:14.532598    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-jtmt6"
Sep 11 20:10:14.800793 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:14.800770    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-jtmt6"]
Sep 11 20:10:14.802445 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:10:14.802420    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f2ec85_7849_4143_9b55_65fe0133f662.slice/crio-fa1477ba04e413bdf0b73242410b997a64c7f9eb39805e2e50b1998660808bbe WatchSource:0}: Error finding container fa1477ba04e413bdf0b73242410b997a64c7f9eb39805e2e50b1998660808bbe: Status 404 returned error can't find the container with id fa1477ba04e413bdf0b73242410b997a64c7f9eb39805e2e50b1998660808bbe
Sep 11 20:10:15.101771 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:15.101689    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-jtmt6" event={"ID":"76f2ec85-7849-4143-9b55-65fe0133f662","Type":"ContainerStarted","Data":"fa1477ba04e413bdf0b73242410b997a64c7f9eb39805e2e50b1998660808bbe"}
Sep 11 20:10:16.108199 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:16.108156    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-jtmt6" event={"ID":"76f2ec85-7849-4143-9b55-65fe0133f662","Type":"ContainerStarted","Data":"92938a63bc232e29415c990cd7148cf884e7f26caf4d24423318e625a036c0e0"}
Sep 11 20:10:16.108619 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:16.108248    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-jtmt6"
Sep 11 20:10:16.124062 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:16.124020    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-jtmt6" podStartSLOduration=0.899279514 podStartE2EDuration="2.124001877s" totalImagesPullingTime="1.224722363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:10:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:10:16.123403344 +0000 UTC m=+586.115565614" watchObservedRunningTime="2026-09-11 20:10:16.124001877 +0000 UTC m=+586.116164146"
Sep 11 20:10:27.116692 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:27.116661    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-jtmt6"
Sep 11 20:10:38.705516 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:38.705464    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:10:47.616937 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:47.616904    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:10:56.798724 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:10:56.798646    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:11:01.510278 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:01.510243    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:11:05.608366 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:05.608332    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:11:24.393614 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.393579    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"]
Sep 11 20:11:24.397018 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.396998    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"
Sep 11 20:11:24.399838 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.399815    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-j8965\""
Sep 11 20:11:24.399952 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.399859    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 20:11:24.400931 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.400911    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 20:11:24.401604 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.401567    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 11 20:11:24.412359 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.412328    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"]
Sep 11 20:11:24.484771 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.484736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d41da7e7-8387-4e8e-b5cc-cc45be55614a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7wqzj\" (UID: \"d41da7e7-8387-4e8e-b5cc-cc45be55614a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"
Sep 11 20:11:24.484771 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.484782    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d41da7e7-8387-4e8e-b5cc-cc45be55614a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7wqzj\" (UID: \"d41da7e7-8387-4e8e-b5cc-cc45be55614a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"
Sep 11 20:11:24.484997 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.484813    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stv2l\" (UniqueName: \"kubernetes.io/projected/d41da7e7-8387-4e8e-b5cc-cc45be55614a-kube-api-access-stv2l\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7wqzj\" (UID: \"d41da7e7-8387-4e8e-b5cc-cc45be55614a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"
Sep 11 20:11:24.484997 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.484837    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d41da7e7-8387-4e8e-b5cc-cc45be55614a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7wqzj\" (UID: \"d41da7e7-8387-4e8e-b5cc-cc45be55614a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"
Sep 11 20:11:24.484997 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.484857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d41da7e7-8387-4e8e-b5cc-cc45be55614a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7wqzj\" (UID: \"d41da7e7-8387-4e8e-b5cc-cc45be55614a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"
Sep 11 20:11:24.484997 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.484873    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d41da7e7-8387-4e8e-b5cc-cc45be55614a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7wqzj\" (UID: \"d41da7e7-8387-4e8e-b5cc-cc45be55614a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"
Sep 11 20:11:24.585861 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.585829    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stv2l\" (UniqueName: \"kubernetes.io/projected/d41da7e7-8387-4e8e-b5cc-cc45be55614a-kube-api-access-stv2l\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7wqzj\" (UID: \"d41da7e7-8387-4e8e-b5cc-cc45be55614a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"
Sep 11 20:11:24.585861 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.585868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d41da7e7-8387-4e8e-b5cc-cc45be55614a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7wqzj\" (UID: \"d41da7e7-8387-4e8e-b5cc-cc45be55614a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"
Sep 11 20:11:24.586123 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.585894    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d41da7e7-8387-4e8e-b5cc-cc45be55614a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7wqzj\" (UID: \"d41da7e7-8387-4e8e-b5cc-cc45be55614a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"
Sep 11 20:11:24.586123 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.585911    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d41da7e7-8387-4e8e-b5cc-cc45be55614a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7wqzj\" (UID: \"d41da7e7-8387-4e8e-b5cc-cc45be55614a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"
Sep 11 20:11:24.586123 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.585946    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d41da7e7-8387-4e8e-b5cc-cc45be55614a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7wqzj\" (UID: \"d41da7e7-8387-4e8e-b5cc-cc45be55614a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"
Sep 11 20:11:24.586123 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.585970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d41da7e7-8387-4e8e-b5cc-cc45be55614a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7wqzj\" (UID: \"d41da7e7-8387-4e8e-b5cc-cc45be55614a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"
Sep 11 20:11:24.586326 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.586281    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d41da7e7-8387-4e8e-b5cc-cc45be55614a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7wqzj\" (UID: \"d41da7e7-8387-4e8e-b5cc-cc45be55614a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"
Sep 11 20:11:24.586387 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.586335    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d41da7e7-8387-4e8e-b5cc-cc45be55614a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7wqzj\" (UID: \"d41da7e7-8387-4e8e-b5cc-cc45be55614a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"
Sep 11 20:11:24.586487 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.586468    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d41da7e7-8387-4e8e-b5cc-cc45be55614a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7wqzj\" (UID: \"d41da7e7-8387-4e8e-b5cc-cc45be55614a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"
Sep 11 20:11:24.588205 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.588186    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d41da7e7-8387-4e8e-b5cc-cc45be55614a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7wqzj\" (UID: \"d41da7e7-8387-4e8e-b5cc-cc45be55614a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"
Sep 11 20:11:24.588462 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.588444    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d41da7e7-8387-4e8e-b5cc-cc45be55614a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7wqzj\" (UID: \"d41da7e7-8387-4e8e-b5cc-cc45be55614a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"
Sep 11 20:11:24.596371 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.596346    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stv2l\" (UniqueName: \"kubernetes.io/projected/d41da7e7-8387-4e8e-b5cc-cc45be55614a-kube-api-access-stv2l\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7wqzj\" (UID: \"d41da7e7-8387-4e8e-b5cc-cc45be55614a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"
Sep 11 20:11:24.710082 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.710046    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"
Sep 11 20:11:24.983191 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.983157    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"]
Sep 11 20:11:24.984899 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:11:24.984872    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd41da7e7_8387_4e8e_b5cc_cc45be55614a.slice/crio-be0a0052f07fe59d0ee4faf0a7bcf32f90330dd50e55e1c73da61171c788d940 WatchSource:0}: Error finding container be0a0052f07fe59d0ee4faf0a7bcf32f90330dd50e55e1c73da61171c788d940: Status 404 returned error can't find the container with id be0a0052f07fe59d0ee4faf0a7bcf32f90330dd50e55e1c73da61171c788d940
Sep 11 20:11:24.986195 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:24.986176    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 20:11:25.449163 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:25.449120    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj" event={"ID":"d41da7e7-8387-4e8e-b5cc-cc45be55614a","Type":"ContainerStarted","Data":"be0a0052f07fe59d0ee4faf0a7bcf32f90330dd50e55e1c73da61171c788d940"}
Sep 11 20:11:26.109984 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:26.109948    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:11:26.456041 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:26.456010    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj" event={"ID":"d41da7e7-8387-4e8e-b5cc-cc45be55614a","Type":"ContainerStarted","Data":"64ccbbe96ab4765d61357b85eee540be1ba4f13bfe615b02e6855a6b7870f94d"}
Sep 11 20:11:26.475446 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:26.475393    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj" podStartSLOduration=1.481709243 podStartE2EDuration="2.475376726s" totalImagesPullingTime="993.667483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:11:26.47362938 +0000 UTC m=+656.465791650" watchObservedRunningTime="2026-09-11 20:11:26.475376726 +0000 UTC m=+656.467538996"
Sep 11 20:11:30.490730 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:30.490697    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"]
Sep 11 20:11:30.494007 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:30.493986    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"
Sep 11 20:11:30.496530 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:30.496508    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 20:11:30.504029 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:30.504006    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"]
Sep 11 20:11:30.638919 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:30.638884    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5d71cd2-848a-49f5-a492-16b652abb1f3-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9\" (UID: \"d5d71cd2-848a-49f5-a492-16b652abb1f3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"
Sep 11 20:11:30.639108 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:30.638928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgk4h\" (UniqueName: \"kubernetes.io/projected/d5d71cd2-848a-49f5-a492-16b652abb1f3-kube-api-access-sgk4h\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9\" (UID: \"d5d71cd2-848a-49f5-a492-16b652abb1f3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"
Sep 11 20:11:30.639108 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:30.638955    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5d71cd2-848a-49f5-a492-16b652abb1f3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9\" (UID: \"d5d71cd2-848a-49f5-a492-16b652abb1f3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"
Sep 11 20:11:30.639108 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:30.638978    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d71cd2-848a-49f5-a492-16b652abb1f3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9\" (UID: \"d5d71cd2-848a-49f5-a492-16b652abb1f3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"
Sep 11 20:11:30.639108 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:30.639007    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5d71cd2-848a-49f5-a492-16b652abb1f3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9\" (UID: \"d5d71cd2-848a-49f5-a492-16b652abb1f3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"
Sep 11 20:11:30.639108 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:30.639088    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5d71cd2-848a-49f5-a492-16b652abb1f3-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9\" (UID: \"d5d71cd2-848a-49f5-a492-16b652abb1f3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"
Sep 11 20:11:30.739916 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:30.739883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5d71cd2-848a-49f5-a492-16b652abb1f3-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9\" (UID: \"d5d71cd2-848a-49f5-a492-16b652abb1f3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"
Sep 11 20:11:30.740089 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:30.739919    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgk4h\" (UniqueName: \"kubernetes.io/projected/d5d71cd2-848a-49f5-a492-16b652abb1f3-kube-api-access-sgk4h\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9\" (UID: \"d5d71cd2-848a-49f5-a492-16b652abb1f3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"
Sep 11 20:11:30.740089 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:30.739942    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5d71cd2-848a-49f5-a492-16b652abb1f3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9\" (UID: \"d5d71cd2-848a-49f5-a492-16b652abb1f3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"
Sep 11 20:11:30.740089 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:30.739961    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d71cd2-848a-49f5-a492-16b652abb1f3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9\" (UID: \"d5d71cd2-848a-49f5-a492-16b652abb1f3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"
Sep 11 20:11:30.740089 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:30.739978    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5d71cd2-848a-49f5-a492-16b652abb1f3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9\" (UID: \"d5d71cd2-848a-49f5-a492-16b652abb1f3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"
Sep 11 20:11:30.740089 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:30.740012    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5d71cd2-848a-49f5-a492-16b652abb1f3-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9\" (UID: \"d5d71cd2-848a-49f5-a492-16b652abb1f3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"
Sep 11 20:11:30.740377 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:30.740351    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5d71cd2-848a-49f5-a492-16b652abb1f3-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9\" (UID: \"d5d71cd2-848a-49f5-a492-16b652abb1f3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"
Sep 11 20:11:30.740440 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:30.740389    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5d71cd2-848a-49f5-a492-16b652abb1f3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9\" (UID: \"d5d71cd2-848a-49f5-a492-16b652abb1f3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"
Sep 11 20:11:30.740478 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:30.740462    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5d71cd2-848a-49f5-a492-16b652abb1f3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9\" (UID: \"d5d71cd2-848a-49f5-a492-16b652abb1f3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"
Sep 11 20:11:30.742267 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:30.742213    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5d71cd2-848a-49f5-a492-16b652abb1f3-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9\" (UID: \"d5d71cd2-848a-49f5-a492-16b652abb1f3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"
Sep 11 20:11:30.742534 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:30.742514    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d71cd2-848a-49f5-a492-16b652abb1f3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9\" (UID: \"d5d71cd2-848a-49f5-a492-16b652abb1f3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"
Sep 11 20:11:30.748075 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:30.748055    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgk4h\" (UniqueName: \"kubernetes.io/projected/d5d71cd2-848a-49f5-a492-16b652abb1f3-kube-api-access-sgk4h\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9\" (UID: \"d5d71cd2-848a-49f5-a492-16b652abb1f3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"
Sep 11 20:11:30.804202 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:30.804180    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"
Sep 11 20:11:31.082014 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:31.081979    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"]
Sep 11 20:11:31.083238 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:11:31.083215    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5d71cd2_848a_49f5_a492_16b652abb1f3.slice/crio-b83c4ec19166c6c3da8a4e96b09aa060244508c122ef9779556af2b521f39548 WatchSource:0}: Error finding container b83c4ec19166c6c3da8a4e96b09aa060244508c122ef9779556af2b521f39548: Status 404 returned error can't find the container with id b83c4ec19166c6c3da8a4e96b09aa060244508c122ef9779556af2b521f39548
Sep 11 20:11:31.481532 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:31.481471    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9" event={"ID":"d5d71cd2-848a-49f5-a492-16b652abb1f3","Type":"ContainerStarted","Data":"80532151baa269fe23327f81aeea020072434817b9d6d44084a5bba160bee891"}
Sep 11 20:11:31.481708 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:31.481545    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9" event={"ID":"d5d71cd2-848a-49f5-a492-16b652abb1f3","Type":"ContainerStarted","Data":"b83c4ec19166c6c3da8a4e96b09aa060244508c122ef9779556af2b521f39548"}
Sep 11 20:11:31.501953 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:31.501897    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9" podStartSLOduration=1.303512406 podStartE2EDuration="1.501880135s" totalImagesPullingTime="198.367729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:11:31.49934663 +0000 UTC m=+661.491508900" watchObservedRunningTime="2026-09-11 20:11:31.501880135 +0000 UTC m=+661.494042404"
Sep 11 20:11:32.805373 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:32.805337    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:11:34.710849 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:34.710818    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"
Sep 11 20:11:34.711193 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:34.710894    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"
Sep 11 20:11:34.723174 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:34.723152    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"
Sep 11 20:11:35.511112 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:35.511082    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7wqzj"
Sep 11 20:11:40.805375 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:40.805340    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"
Sep 11 20:11:40.805806 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:40.805394    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"
Sep 11 20:11:40.817750 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:40.817722    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"
Sep 11 20:11:41.539100 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:11:41.539073    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9"
Sep 11 20:12:36.423106 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:36.423065    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b84cff79-88hf6"]
Sep 11 20:12:36.426506 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:36.426468    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b84cff79-88hf6"
Sep 11 20:12:36.429985 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:36.429966    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 20:12:36.430085 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:36.429988    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5tkpk\""
Sep 11 20:12:36.446649 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:36.446618    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b84cff79-88hf6"]
Sep 11 20:12:36.566987 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:36.566948    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1f9e71e-d948-4142-a345-5e8231aabf2d-tls-cert\") pod \"authorino-5b84cff79-88hf6\" (UID: \"a1f9e71e-d948-4142-a345-5e8231aabf2d\") " pod="kuadrant-system/authorino-5b84cff79-88hf6"
Sep 11 20:12:36.567148 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:36.566999    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg4rr\" (UniqueName: \"kubernetes.io/projected/a1f9e71e-d948-4142-a345-5e8231aabf2d-kube-api-access-wg4rr\") pod \"authorino-5b84cff79-88hf6\" (UID: \"a1f9e71e-d948-4142-a345-5e8231aabf2d\") " pod="kuadrant-system/authorino-5b84cff79-88hf6"
Sep 11 20:12:36.667982 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:36.667947    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1f9e71e-d948-4142-a345-5e8231aabf2d-tls-cert\") pod \"authorino-5b84cff79-88hf6\" (UID: \"a1f9e71e-d948-4142-a345-5e8231aabf2d\") " pod="kuadrant-system/authorino-5b84cff79-88hf6"
Sep 11 20:12:36.668133 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:36.667996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg4rr\" (UniqueName: \"kubernetes.io/projected/a1f9e71e-d948-4142-a345-5e8231aabf2d-kube-api-access-wg4rr\") pod \"authorino-5b84cff79-88hf6\" (UID: \"a1f9e71e-d948-4142-a345-5e8231aabf2d\") " pod="kuadrant-system/authorino-5b84cff79-88hf6"
Sep 11 20:12:36.670461 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:36.670435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1f9e71e-d948-4142-a345-5e8231aabf2d-tls-cert\") pod \"authorino-5b84cff79-88hf6\" (UID: \"a1f9e71e-d948-4142-a345-5e8231aabf2d\") " pod="kuadrant-system/authorino-5b84cff79-88hf6"
Sep 11 20:12:36.680377 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:36.680321    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg4rr\" (UniqueName: \"kubernetes.io/projected/a1f9e71e-d948-4142-a345-5e8231aabf2d-kube-api-access-wg4rr\") pod \"authorino-5b84cff79-88hf6\" (UID: \"a1f9e71e-d948-4142-a345-5e8231aabf2d\") " pod="kuadrant-system/authorino-5b84cff79-88hf6"
Sep 11 20:12:36.736578 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:36.736552    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b84cff79-88hf6"
Sep 11 20:12:37.005971 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:37.005945    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b84cff79-88hf6"]
Sep 11 20:12:37.007721 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:12:37.007690    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1f9e71e_d948_4142_a345_5e8231aabf2d.slice/crio-47373c8105847e67caeb42116110a77d27763352274c2d3b104a4eb0a5112e9b WatchSource:0}: Error finding container 47373c8105847e67caeb42116110a77d27763352274c2d3b104a4eb0a5112e9b: Status 404 returned error can't find the container with id 47373c8105847e67caeb42116110a77d27763352274c2d3b104a4eb0a5112e9b
Sep 11 20:12:37.814314 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:37.814279    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b84cff79-88hf6" event={"ID":"a1f9e71e-d948-4142-a345-5e8231aabf2d","Type":"ContainerStarted","Data":"ba8dc7f10bc55760aae39927eb116bd61852d012dc28cc3d2eaeb3b905d3ab27"}
Sep 11 20:12:37.814763 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:37.814327    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b84cff79-88hf6" event={"ID":"a1f9e71e-d948-4142-a345-5e8231aabf2d","Type":"ContainerStarted","Data":"47373c8105847e67caeb42116110a77d27763352274c2d3b104a4eb0a5112e9b"}
Sep 11 20:12:56.352904 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:56.352841    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5b84cff79-88hf6" podStartSLOduration=19.705509323 podStartE2EDuration="20.352819266s" totalImagesPullingTime="647.309943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:12:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:12:37.831687167 +0000 UTC m=+727.823849450" watchObservedRunningTime="2026-09-11 20:12:56.352819266 +0000 UTC m=+746.344981529"
Sep 11 20:12:56.354740 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:56.354707    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7565-a21f5c"]
Sep 11 20:12:56.357814 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:56.357795    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7565-a21f5c"
Sep 11 20:12:56.360355 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:56.360335    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-94fzf\""
Sep 11 20:12:56.365819 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:56.365799    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7565-a21f5c"]
Sep 11 20:12:56.423029 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:56.422996    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g95l6\" (UniqueName: \"kubernetes.io/projected/9344cc07-947f-4a9e-b04b-469f2d1ffb39-kube-api-access-g95l6\") pod \"test-curl-7565-a21f5c\" (UID: \"9344cc07-947f-4a9e-b04b-469f2d1ffb39\") " pod="opendatahub/test-curl-7565-a21f5c"
Sep 11 20:12:56.524136 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:56.524096    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g95l6\" (UniqueName: \"kubernetes.io/projected/9344cc07-947f-4a9e-b04b-469f2d1ffb39-kube-api-access-g95l6\") pod \"test-curl-7565-a21f5c\" (UID: \"9344cc07-947f-4a9e-b04b-469f2d1ffb39\") " pod="opendatahub/test-curl-7565-a21f5c"
Sep 11 20:12:56.532735 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:56.532709    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g95l6\" (UniqueName: \"kubernetes.io/projected/9344cc07-947f-4a9e-b04b-469f2d1ffb39-kube-api-access-g95l6\") pod \"test-curl-7565-a21f5c\" (UID: \"9344cc07-947f-4a9e-b04b-469f2d1ffb39\") " pod="opendatahub/test-curl-7565-a21f5c"
Sep 11 20:12:56.668765 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:56.668685    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7565-a21f5c"
Sep 11 20:12:56.945433 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:56.945412    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7565-a21f5c"]
Sep 11 20:12:56.947305 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:12:56.947280    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9344cc07_947f_4a9e_b04b_469f2d1ffb39.slice/crio-3770754ca270aee2df275550e26c63d459f85770c4c8e1e15d1719c4d74bdecf WatchSource:0}: Error finding container 3770754ca270aee2df275550e26c63d459f85770c4c8e1e15d1719c4d74bdecf: Status 404 returned error can't find the container with id 3770754ca270aee2df275550e26c63d459f85770c4c8e1e15d1719c4d74bdecf
Sep 11 20:12:57.922753 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:57.922708    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7565-a21f5c" event={"ID":"9344cc07-947f-4a9e-b04b-469f2d1ffb39","Type":"ContainerStarted","Data":"3770754ca270aee2df275550e26c63d459f85770c4c8e1e15d1719c4d74bdecf"}
Sep 11 20:12:58.539105 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:58.539069    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:12:59.869270 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:59.869237    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx"]
Sep 11 20:12:59.872216 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:59.872195    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx"
Sep 11 20:12:59.875013 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:59.874995    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b3e1dd-dockercfg-25n9v\""
Sep 11 20:12:59.875097 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:59.875070    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b3e1dd\""
Sep 11 20:12:59.886243 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:59.886218    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx"]
Sep 11 20:12:59.936803 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:59.936765    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7565-a21f5c" event={"ID":"9344cc07-947f-4a9e-b04b-469f2d1ffb39","Type":"ContainerStarted","Data":"c4dfc988e4f169382da6a5c52d20fba984356ac7e88215d10150a90365d6863b"}
Sep 11 20:12:59.951699 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:59.951658    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7565-a21f5c" podStartSLOduration=1.684280499 podStartE2EDuration="3.951643825s" totalImagesPullingTime="2.267363326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:12:59.950915851 +0000 UTC m=+749.943078120" watchObservedRunningTime="2026-09-11 20:12:59.951643825 +0000 UTC m=+749.943806094"
Sep 11 20:12:59.960202 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:59.960178    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxzlw\" (UniqueName: \"kubernetes.io/projected/a0588500-5e82-4cfe-9a88-b5795b7b03a5-kube-api-access-lxzlw\") pod \"payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx\" (UID: \"a0588500-5e82-4cfe-9a88-b5795b7b03a5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx"
Sep 11 20:12:59.960281 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:12:59.960235    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0588500-5e82-4cfe-9a88-b5795b7b03a5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx\" (UID: \"a0588500-5e82-4cfe-9a88-b5795b7b03a5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx"
Sep 11 20:13:00.061060 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.061020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxzlw\" (UniqueName: \"kubernetes.io/projected/a0588500-5e82-4cfe-9a88-b5795b7b03a5-kube-api-access-lxzlw\") pod \"payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx\" (UID: \"a0588500-5e82-4cfe-9a88-b5795b7b03a5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx"
Sep 11 20:13:00.061231 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.061097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0588500-5e82-4cfe-9a88-b5795b7b03a5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx\" (UID: \"a0588500-5e82-4cfe-9a88-b5795b7b03a5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx"
Sep 11 20:13:00.061778 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.061759    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0588500-5e82-4cfe-9a88-b5795b7b03a5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx\" (UID: \"a0588500-5e82-4cfe-9a88-b5795b7b03a5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx"
Sep 11 20:13:00.069726 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.069697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxzlw\" (UniqueName: \"kubernetes.io/projected/a0588500-5e82-4cfe-9a88-b5795b7b03a5-kube-api-access-lxzlw\") pod \"payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx\" (UID: \"a0588500-5e82-4cfe-9a88-b5795b7b03a5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx"
Sep 11 20:13:00.182625 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.182551    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx"
Sep 11 20:13:00.461856 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.461832    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx"]
Sep 11 20:13:00.464081 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:13:00.464053    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0588500_5e82_4cfe_9a88_b5795b7b03a5.slice/crio-6c31c6e21d223154d35cd5b7a5017a6fdb54c05ea6bc5ce4fb36baa55a73e1d6 WatchSource:0}: Error finding container 6c31c6e21d223154d35cd5b7a5017a6fdb54c05ea6bc5ce4fb36baa55a73e1d6: Status 404 returned error can't find the container with id 6c31c6e21d223154d35cd5b7a5017a6fdb54c05ea6bc5ce4fb36baa55a73e1d6
Sep 11 20:13:00.511939 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.511916    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7565-a21f5c"]
Sep 11 20:13:00.519059 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.519036    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7565-a21f5c"]
Sep 11 20:13:00.629847 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.629812    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7565-a20e7d"]
Sep 11 20:13:00.630141 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.630128    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9344cc07-947f-4a9e-b04b-469f2d1ffb39" containerName="test-curl-7565-a21f5c"
Sep 11 20:13:00.630208 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.630142    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9344cc07-947f-4a9e-b04b-469f2d1ffb39" containerName="test-curl-7565-a21f5c"
Sep 11 20:13:00.630208 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.630203    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9344cc07-947f-4a9e-b04b-469f2d1ffb39" containerName="test-curl-7565-a21f5c"
Sep 11 20:13:00.633106 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.633082    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7565-a20e7d"
Sep 11 20:13:00.641082 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.641059    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7565-a20e7d"]
Sep 11 20:13:00.665109 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.665081    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkf7g\" (UniqueName: \"kubernetes.io/projected/435ff654-1f88-489e-90df-a18c422e98fe-kube-api-access-gkf7g\") pod \"test-curl-7565-a20e7d\" (UID: \"435ff654-1f88-489e-90df-a18c422e98fe\") " pod="opendatahub/test-curl-7565-a20e7d"
Sep 11 20:13:00.766106 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.766018    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkf7g\" (UniqueName: \"kubernetes.io/projected/435ff654-1f88-489e-90df-a18c422e98fe-kube-api-access-gkf7g\") pod \"test-curl-7565-a20e7d\" (UID: \"435ff654-1f88-489e-90df-a18c422e98fe\") " pod="opendatahub/test-curl-7565-a20e7d"
Sep 11 20:13:00.785982 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.785951    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkf7g\" (UniqueName: \"kubernetes.io/projected/435ff654-1f88-489e-90df-a18c422e98fe-kube-api-access-gkf7g\") pod \"test-curl-7565-a20e7d\" (UID: \"435ff654-1f88-489e-90df-a18c422e98fe\") " pod="opendatahub/test-curl-7565-a20e7d"
Sep 11 20:13:00.942624 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.942595    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7565-a20e7d"
Sep 11 20:13:00.944001 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.943974    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx" event={"ID":"a0588500-5e82-4cfe-9a88-b5795b7b03a5","Type":"ContainerStarted","Data":"c087ffb4b2fd2cbe5ae1d4c419cb482b21c942d64fdd667d1af7bd971de851ee"}
Sep 11 20:13:00.944202 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:13:00.944182    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7565-a21f5c" container="test-curl-7565-a21f5c"
Sep 11 20:13:00.944252 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.944195    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx"
Sep 11 20:13:00.944288 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.944250    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx" event={"ID":"a0588500-5e82-4cfe-9a88-b5795b7b03a5","Type":"ContainerStarted","Data":"6c31c6e21d223154d35cd5b7a5017a6fdb54c05ea6bc5ce4fb36baa55a73e1d6"}
Sep 11 20:13:00.982179 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.982146    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl"]
Sep 11 20:13:00.983194 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.983155    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx" podStartSLOduration=1.9831398770000002 podStartE2EDuration="1.983139877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:00.979455162 +0000 UTC m=+750.971617436" watchObservedRunningTime="2026-09-11 20:13:00.983139877 +0000 UTC m=+750.975302146"
Sep 11 20:13:00.985844 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.985748    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl"
Sep 11 20:13:00.988712 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.988690    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-22fc84\""
Sep 11 20:13:00.988829 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.988772    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-22fc84-dockercfg-kzcwb\""
Sep 11 20:13:00.998452 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:00.997998    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl"]
Sep 11 20:13:01.069740 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.069666    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d36169a-7242-4916-9307-3deee76d24b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl\" (UID: \"6d36169a-7242-4916-9307-3deee76d24b1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl"
Sep 11 20:13:01.069740 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.069724    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgvhp\" (UniqueName: \"kubernetes.io/projected/6d36169a-7242-4916-9307-3deee76d24b1-kube-api-access-kgvhp\") pod \"payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl\" (UID: \"6d36169a-7242-4916-9307-3deee76d24b1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl"
Sep 11 20:13:01.079367 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.079341    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg"]
Sep 11 20:13:01.082706 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.082689    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg"
Sep 11 20:13:01.094274 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.094255    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg"]
Sep 11 20:13:01.170936 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.170911    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7vcn\" (UniqueName: \"kubernetes.io/projected/d943c471-9182-4add-a825-58deb30b3816-kube-api-access-v7vcn\") pod \"payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg\" (UID: \"d943c471-9182-4add-a825-58deb30b3816\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg"
Sep 11 20:13:01.171049 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.170964    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d36169a-7242-4916-9307-3deee76d24b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl\" (UID: \"6d36169a-7242-4916-9307-3deee76d24b1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl"
Sep 11 20:13:01.171049 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.171026    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgvhp\" (UniqueName: \"kubernetes.io/projected/6d36169a-7242-4916-9307-3deee76d24b1-kube-api-access-kgvhp\") pod \"payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl\" (UID: \"6d36169a-7242-4916-9307-3deee76d24b1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl"
Sep 11 20:13:01.171152 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.171095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d943c471-9182-4add-a825-58deb30b3816-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg\" (UID: \"d943c471-9182-4add-a825-58deb30b3816\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg"
Sep 11 20:13:01.171712 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.171694    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d36169a-7242-4916-9307-3deee76d24b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl\" (UID: \"6d36169a-7242-4916-9307-3deee76d24b1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl"
Sep 11 20:13:01.179795 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.179772    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgvhp\" (UniqueName: \"kubernetes.io/projected/6d36169a-7242-4916-9307-3deee76d24b1-kube-api-access-kgvhp\") pod \"payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl\" (UID: \"6d36169a-7242-4916-9307-3deee76d24b1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl"
Sep 11 20:13:01.230811 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.230788    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7565-a20e7d"]
Sep 11 20:13:01.232446 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:13:01.232424    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod435ff654_1f88_489e_90df_a18c422e98fe.slice/crio-156aa5b658afc30dc309a0e484ef3be217d3733fd31ec37f1ad7a45b497e4ba2 WatchSource:0}: Error finding container 156aa5b658afc30dc309a0e484ef3be217d3733fd31ec37f1ad7a45b497e4ba2: Status 404 returned error can't find the container with id 156aa5b658afc30dc309a0e484ef3be217d3733fd31ec37f1ad7a45b497e4ba2
Sep 11 20:13:01.271582 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.271548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d943c471-9182-4add-a825-58deb30b3816-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg\" (UID: \"d943c471-9182-4add-a825-58deb30b3816\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg"
Sep 11 20:13:01.271687 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.271647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7vcn\" (UniqueName: \"kubernetes.io/projected/d943c471-9182-4add-a825-58deb30b3816-kube-api-access-v7vcn\") pod \"payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg\" (UID: \"d943c471-9182-4add-a825-58deb30b3816\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg"
Sep 11 20:13:01.272287 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.272262    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d943c471-9182-4add-a825-58deb30b3816-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg\" (UID: \"d943c471-9182-4add-a825-58deb30b3816\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg"
Sep 11 20:13:01.281728 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.281709    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7vcn\" (UniqueName: \"kubernetes.io/projected/d943c471-9182-4add-a825-58deb30b3816-kube-api-access-v7vcn\") pod \"payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg\" (UID: \"d943c471-9182-4add-a825-58deb30b3816\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg"
Sep 11 20:13:01.297921 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.297900    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl"
Sep 11 20:13:01.394955 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.394930    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg"
Sep 11 20:13:01.568449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.568416    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl"]
Sep 11 20:13:01.570129 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:13:01.570103    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d36169a_7242_4916_9307_3deee76d24b1.slice/crio-a9b8d701479e7898e7df9dd072a910582366e7bc800303e2cf3807ce535cc0d9 WatchSource:0}: Error finding container a9b8d701479e7898e7df9dd072a910582366e7bc800303e2cf3807ce535cc0d9: Status 404 returned error can't find the container with id a9b8d701479e7898e7df9dd072a910582366e7bc800303e2cf3807ce535cc0d9
Sep 11 20:13:01.667662 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.667625    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg"]
Sep 11 20:13:01.669376 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:13:01.669351    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd943c471_9182_4add_a825_58deb30b3816.slice/crio-8320782a0b248446f8e46d11cee8b7298c0bd7659f855b035a805b1fb05313da WatchSource:0}: Error finding container 8320782a0b248446f8e46d11cee8b7298c0bd7659f855b035a805b1fb05313da: Status 404 returned error can't find the container with id 8320782a0b248446f8e46d11cee8b7298c0bd7659f855b035a805b1fb05313da
Sep 11 20:13:01.953105 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.953071    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl" event={"ID":"6d36169a-7242-4916-9307-3deee76d24b1","Type":"ContainerStarted","Data":"e9a266b3e511b08bd9882b7f6b563c475af84d80024d2bce7f1695ce586c28f4"}
Sep 11 20:13:01.953586 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.953147    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl" event={"ID":"6d36169a-7242-4916-9307-3deee76d24b1","Type":"ContainerStarted","Data":"a9b8d701479e7898e7df9dd072a910582366e7bc800303e2cf3807ce535cc0d9"}
Sep 11 20:13:01.953586 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.953363    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl"
Sep 11 20:13:01.955429 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.955384    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg" event={"ID":"d943c471-9182-4add-a825-58deb30b3816","Type":"ContainerStarted","Data":"a0afecef9b4ea4f5f28969b7cebf6cc6ddb0645e5f8da6b01a74d3a76e1e31b6"}
Sep 11 20:13:01.955429 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.955431    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg" event={"ID":"d943c471-9182-4add-a825-58deb30b3816","Type":"ContainerStarted","Data":"8320782a0b248446f8e46d11cee8b7298c0bd7659f855b035a805b1fb05313da"}
Sep 11 20:13:01.955738 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.955522    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg"
Sep 11 20:13:01.957579 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.957523    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7565-a20e7d" event={"ID":"435ff654-1f88-489e-90df-a18c422e98fe","Type":"ContainerStarted","Data":"b5f068a3db6f21cb0b26a13ef72d3304a7c504e4c55721727f3145f2abf4ad4a"}
Sep 11 20:13:01.957579 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.957555    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7565-a20e7d" event={"ID":"435ff654-1f88-489e-90df-a18c422e98fe","Type":"ContainerStarted","Data":"156aa5b658afc30dc309a0e484ef3be217d3733fd31ec37f1ad7a45b497e4ba2"}
Sep 11 20:13:01.957749 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.957612    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7565-a21f5c" podUID="9344cc07-947f-4a9e-b04b-469f2d1ffb39" containerName="test-curl-7565-a21f5c" containerID="cri-o://c4dfc988e4f169382da6a5c52d20fba984356ac7e88215d10150a90365d6863b" gracePeriod=2
Sep 11 20:13:01.976784 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.976736    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl" podStartSLOduration=1.9767213350000001 podStartE2EDuration="1.976721335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:01.974967435 +0000 UTC m=+751.967129707" watchObservedRunningTime="2026-09-11 20:13:01.976721335 +0000 UTC m=+751.968883604"
Sep 11 20:13:01.977056 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:13:01.977017    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7565-a21f5c\" is forbidden: User \"system:node:ip-10-0-128-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-209.ec2.internal' and this object" podUID="9344cc07-947f-4a9e-b04b-469f2d1ffb39" pod="opendatahub/test-curl-7565-a21f5c"
Sep 11 20:13:01.995472 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:01.995414    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7565-a20e7d" podStartSLOduration=1.9953926530000001 podStartE2EDuration="1.995392653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:01.992189906 +0000 UTC m=+751.984352159" watchObservedRunningTime="2026-09-11 20:13:01.995392653 +0000 UTC m=+751.987554925"
Sep 11 20:13:02.011270 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.011212    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg" podStartSLOduration=1.011196813 podStartE2EDuration="1.011196813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:02.008390507 +0000 UTC m=+752.000552779" watchObservedRunningTime="2026-09-11 20:13:02.011196813 +0000 UTC m=+752.003359082"
Sep 11 20:13:02.080355 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.080320    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt"]
Sep 11 20:13:02.083830 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.083814    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt"
Sep 11 20:13:02.086337 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:13:02.086306    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7565-a21f5c\" is forbidden: User \"system:node:ip-10-0-128-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-209.ec2.internal' and this object" podUID="9344cc07-947f-4a9e-b04b-469f2d1ffb39" pod="opendatahub/test-curl-7565-a21f5c"
Sep 11 20:13:02.086581 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.086561    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-9d9abb\""
Sep 11 20:13:02.086820 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.086799    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-9d9abb-dockercfg-f8dbh\""
Sep 11 20:13:02.094138 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.094117    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt"]
Sep 11 20:13:02.180242 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.180209    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34faadd9-bb65-4973-8af6-150cdbbd0287-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt\" (UID: \"34faadd9-bb65-4973-8af6-150cdbbd0287\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt"
Sep 11 20:13:02.180404 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.180276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmjzg\" (UniqueName: \"kubernetes.io/projected/34faadd9-bb65-4973-8af6-150cdbbd0287-kube-api-access-bmjzg\") pod \"payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt\" (UID: \"34faadd9-bb65-4973-8af6-150cdbbd0287\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt"
Sep 11 20:13:02.212024 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.211955    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8"]
Sep 11 20:13:02.215312 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.215293    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8"
Sep 11 20:13:02.230589 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.230561    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8"]
Sep 11 20:13:02.248924 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:13:02.248886    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7565-a21f5c\" is forbidden: User \"system:node:ip-10-0-128-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-209.ec2.internal' and this object" podUID="9344cc07-947f-4a9e-b04b-469f2d1ffb39" pod="opendatahub/test-curl-7565-a21f5c"
Sep 11 20:13:02.280987 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.280955    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmjzg\" (UniqueName: \"kubernetes.io/projected/34faadd9-bb65-4973-8af6-150cdbbd0287-kube-api-access-bmjzg\") pod \"payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt\" (UID: \"34faadd9-bb65-4973-8af6-150cdbbd0287\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt"
Sep 11 20:13:02.281148 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.281029    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98d64d0d-e82c-4071-8239-4d785b143bec-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8\" (UID: \"98d64d0d-e82c-4071-8239-4d785b143bec\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8"
Sep 11 20:13:02.281148 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.281050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34faadd9-bb65-4973-8af6-150cdbbd0287-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt\" (UID: \"34faadd9-bb65-4973-8af6-150cdbbd0287\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt"
Sep 11 20:13:02.281148 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.281069    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z9sc\" (UniqueName: \"kubernetes.io/projected/98d64d0d-e82c-4071-8239-4d785b143bec-kube-api-access-6z9sc\") pod \"payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8\" (UID: \"98d64d0d-e82c-4071-8239-4d785b143bec\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8"
Sep 11 20:13:02.281721 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.281698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34faadd9-bb65-4973-8af6-150cdbbd0287-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt\" (UID: \"34faadd9-bb65-4973-8af6-150cdbbd0287\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt"
Sep 11 20:13:02.291395 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.291374    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmjzg\" (UniqueName: \"kubernetes.io/projected/34faadd9-bb65-4973-8af6-150cdbbd0287-kube-api-access-bmjzg\") pod \"payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt\" (UID: \"34faadd9-bb65-4973-8af6-150cdbbd0287\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt"
Sep 11 20:13:02.381800 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.381762    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98d64d0d-e82c-4071-8239-4d785b143bec-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8\" (UID: \"98d64d0d-e82c-4071-8239-4d785b143bec\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8"
Sep 11 20:13:02.381800 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.381808    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z9sc\" (UniqueName: \"kubernetes.io/projected/98d64d0d-e82c-4071-8239-4d785b143bec-kube-api-access-6z9sc\") pod \"payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8\" (UID: \"98d64d0d-e82c-4071-8239-4d785b143bec\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8"
Sep 11 20:13:02.382488 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.382466    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98d64d0d-e82c-4071-8239-4d785b143bec-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8\" (UID: \"98d64d0d-e82c-4071-8239-4d785b143bec\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8"
Sep 11 20:13:02.390693 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.390669    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z9sc\" (UniqueName: \"kubernetes.io/projected/98d64d0d-e82c-4071-8239-4d785b143bec-kube-api-access-6z9sc\") pod \"payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8\" (UID: \"98d64d0d-e82c-4071-8239-4d785b143bec\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8"
Sep 11 20:13:02.395089 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.395065    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt"
Sep 11 20:13:02.525454 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.525364    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8"
Sep 11 20:13:02.642851 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.642774    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7565-a20e7d"]
Sep 11 20:13:02.645221 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.645196    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7565-a20e7d"]
Sep 11 20:13:02.674029 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.674000    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt"]
Sep 11 20:13:02.676314 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:13:02.676284    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34faadd9_bb65_4973_8af6_150cdbbd0287.slice/crio-f4d45ccf5a8606c4eeec487897e46c6c375901882136fa34f3472d7c992dce7b WatchSource:0}: Error finding container f4d45ccf5a8606c4eeec487897e46c6c375901882136fa34f3472d7c992dce7b: Status 404 returned error can't find the container with id f4d45ccf5a8606c4eeec487897e46c6c375901882136fa34f3472d7c992dce7b
Sep 11 20:13:02.826432 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.826392    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8"]
Sep 11 20:13:02.828146 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:13:02.828115    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d64d0d_e82c_4071_8239_4d785b143bec.slice/crio-a3f256414da3fc76fc803cd45017bf438ba3608bc8e1b2e79c9b620698b3ed3e WatchSource:0}: Error finding container a3f256414da3fc76fc803cd45017bf438ba3608bc8e1b2e79c9b620698b3ed3e: Status 404 returned error can't find the container with id a3f256414da3fc76fc803cd45017bf438ba3608bc8e1b2e79c9b620698b3ed3e
Sep 11 20:13:02.966504 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.966453    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt" event={"ID":"34faadd9-bb65-4973-8af6-150cdbbd0287","Type":"ContainerStarted","Data":"a0ddd3dc64078e3cd9e152f942323782f4d999c2c238e7bfd8fba1288ec1fa28"}
Sep 11 20:13:02.966967 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.966532    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt" event={"ID":"34faadd9-bb65-4973-8af6-150cdbbd0287","Type":"ContainerStarted","Data":"f4d45ccf5a8606c4eeec487897e46c6c375901882136fa34f3472d7c992dce7b"}
Sep 11 20:13:02.966967 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.966656    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt"
Sep 11 20:13:02.968356 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.968334    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8" event={"ID":"98d64d0d-e82c-4071-8239-4d785b143bec","Type":"ContainerStarted","Data":"48ee590179c01fce2f58e8b793e50d29a2b041796e7910e19f441361fc7f370e"}
Sep 11 20:13:02.968462 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.968382    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8" event={"ID":"98d64d0d-e82c-4071-8239-4d785b143bec","Type":"ContainerStarted","Data":"a3f256414da3fc76fc803cd45017bf438ba3608bc8e1b2e79c9b620698b3ed3e"}
Sep 11 20:13:02.968675 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:13:02.968659    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7565-a20e7d" container="test-curl-7565-a20e7d"
Sep 11 20:13:02.968799 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.968661    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8"
Sep 11 20:13:02.985358 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:02.985311    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt" podStartSLOduration=0.985299194 podStartE2EDuration="985.299194ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:02.984479614 +0000 UTC m=+752.976641883" watchObservedRunningTime="2026-09-11 20:13:02.985299194 +0000 UTC m=+752.977461463"
Sep 11 20:13:03.013319 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:03.013275    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8" podStartSLOduration=1.013260912 podStartE2EDuration="1.013260912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:03.010910447 +0000 UTC m=+753.003072744" watchObservedRunningTime="2026-09-11 20:13:03.013260912 +0000 UTC m=+753.005423181"
Sep 11 20:13:03.974054 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:03.974018    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7565-a20e7d" podUID="435ff654-1f88-489e-90df-a18c422e98fe" containerName="test-curl-7565-a20e7d" containerID="cri-o://b5f068a3db6f21cb0b26a13ef72d3304a7c504e4c55721727f3145f2abf4ad4a" gracePeriod=2
Sep 11 20:13:03.976752 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:13:03.976726    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7565-a20e7d\" is forbidden: User \"system:node:ip-10-0-128-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-209.ec2.internal' and this object" podUID="435ff654-1f88-489e-90df-a18c422e98fe" pod="opendatahub/test-curl-7565-a20e7d"
Sep 11 20:13:04.126896 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:04.126869    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7565-a21f5c"
Sep 11 20:13:04.129036 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:13:04.129013    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7565-a20e7d\" is forbidden: User \"system:node:ip-10-0-128-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-209.ec2.internal' and this object" podUID="435ff654-1f88-489e-90df-a18c422e98fe" pod="opendatahub/test-curl-7565-a20e7d"
Sep 11 20:13:04.199208 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:04.199181    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g95l6\" (UniqueName: \"kubernetes.io/projected/9344cc07-947f-4a9e-b04b-469f2d1ffb39-kube-api-access-g95l6\") pod \"9344cc07-947f-4a9e-b04b-469f2d1ffb39\" (UID: \"9344cc07-947f-4a9e-b04b-469f2d1ffb39\") "
Sep 11 20:13:04.201293 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:04.201256    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9344cc07-947f-4a9e-b04b-469f2d1ffb39-kube-api-access-g95l6" (OuterVolumeSpecName: "kube-api-access-g95l6") pod "9344cc07-947f-4a9e-b04b-469f2d1ffb39" (UID: "9344cc07-947f-4a9e-b04b-469f2d1ffb39"). InnerVolumeSpecName "kube-api-access-g95l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:13:04.299939 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:04.299878    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g95l6\" (UniqueName: \"kubernetes.io/projected/9344cc07-947f-4a9e-b04b-469f2d1ffb39-kube-api-access-g95l6\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:13:04.441751 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:04.441720    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9344cc07-947f-4a9e-b04b-469f2d1ffb39" path="/var/lib/kubelet/pods/9344cc07-947f-4a9e-b04b-469f2d1ffb39/volumes"
Sep 11 20:13:04.981447 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:04.981415    2580 generic.go:356] "Generic (PLEG): container finished" podID="9344cc07-947f-4a9e-b04b-469f2d1ffb39" containerID="c4dfc988e4f169382da6a5c52d20fba984356ac7e88215d10150a90365d6863b" exitCode=137
Sep 11 20:13:04.982300 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:04.982276    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7565-a21f5c"
Sep 11 20:13:04.982605 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:04.982585    2580 scope.go:117] "RemoveContainer" containerID="c4dfc988e4f169382da6a5c52d20fba984356ac7e88215d10150a90365d6863b"
Sep 11 20:13:04.991353 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:04.991334    2580 scope.go:117] "RemoveContainer" containerID="c4dfc988e4f169382da6a5c52d20fba984356ac7e88215d10150a90365d6863b"
Sep 11 20:13:04.991640 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:13:04.991620    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4dfc988e4f169382da6a5c52d20fba984356ac7e88215d10150a90365d6863b\": container with ID starting with c4dfc988e4f169382da6a5c52d20fba984356ac7e88215d10150a90365d6863b not found: ID does not exist" containerID="c4dfc988e4f169382da6a5c52d20fba984356ac7e88215d10150a90365d6863b"
Sep 11 20:13:04.991712 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:04.991647    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4dfc988e4f169382da6a5c52d20fba984356ac7e88215d10150a90365d6863b"} err="failed to get container status \"c4dfc988e4f169382da6a5c52d20fba984356ac7e88215d10150a90365d6863b\": rpc error: code = NotFound desc = could not find container \"c4dfc988e4f169382da6a5c52d20fba984356ac7e88215d10150a90365d6863b\": container with ID starting with c4dfc988e4f169382da6a5c52d20fba984356ac7e88215d10150a90365d6863b not found: ID does not exist"
Sep 11 20:13:06.161047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:06.161012    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7565-a20e7d"
Sep 11 20:13:06.232213 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:06.232123    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkf7g\" (UniqueName: \"kubernetes.io/projected/435ff654-1f88-489e-90df-a18c422e98fe-kube-api-access-gkf7g\") pod \"435ff654-1f88-489e-90df-a18c422e98fe\" (UID: \"435ff654-1f88-489e-90df-a18c422e98fe\") "
Sep 11 20:13:06.234156 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:06.234123    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/435ff654-1f88-489e-90df-a18c422e98fe-kube-api-access-gkf7g" (OuterVolumeSpecName: "kube-api-access-gkf7g") pod "435ff654-1f88-489e-90df-a18c422e98fe" (UID: "435ff654-1f88-489e-90df-a18c422e98fe"). InnerVolumeSpecName "kube-api-access-gkf7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:13:06.333133 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:06.333098    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkf7g\" (UniqueName: \"kubernetes.io/projected/435ff654-1f88-489e-90df-a18c422e98fe-kube-api-access-gkf7g\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:13:06.442116 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:06.442073    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="435ff654-1f88-489e-90df-a18c422e98fe" path="/var/lib/kubelet/pods/435ff654-1f88-489e-90df-a18c422e98fe/volumes"
Sep 11 20:13:06.996620 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:06.996585    2580 generic.go:356] "Generic (PLEG): container finished" podID="435ff654-1f88-489e-90df-a18c422e98fe" containerID="b5f068a3db6f21cb0b26a13ef72d3304a7c504e4c55721727f3145f2abf4ad4a" exitCode=137
Sep 11 20:13:06.996819 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:06.996637    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7565-a20e7d"
Sep 11 20:13:06.996819 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:06.996689    2580 scope.go:117] "RemoveContainer" containerID="b5f068a3db6f21cb0b26a13ef72d3304a7c504e4c55721727f3145f2abf4ad4a"
Sep 11 20:13:07.007400 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:07.007211    2580 scope.go:117] "RemoveContainer" containerID="b5f068a3db6f21cb0b26a13ef72d3304a7c504e4c55721727f3145f2abf4ad4a"
Sep 11 20:13:07.007591 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:13:07.007569    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5f068a3db6f21cb0b26a13ef72d3304a7c504e4c55721727f3145f2abf4ad4a\": container with ID starting with b5f068a3db6f21cb0b26a13ef72d3304a7c504e4c55721727f3145f2abf4ad4a not found: ID does not exist" containerID="b5f068a3db6f21cb0b26a13ef72d3304a7c504e4c55721727f3145f2abf4ad4a"
Sep 11 20:13:07.007676 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:07.007607    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5f068a3db6f21cb0b26a13ef72d3304a7c504e4c55721727f3145f2abf4ad4a"} err="failed to get container status \"b5f068a3db6f21cb0b26a13ef72d3304a7c504e4c55721727f3145f2abf4ad4a\": rpc error: code = NotFound desc = could not find container \"b5f068a3db6f21cb0b26a13ef72d3304a7c504e4c55721727f3145f2abf4ad4a\": container with ID starting with b5f068a3db6f21cb0b26a13ef72d3304a7c504e4c55721727f3145f2abf4ad4a not found: ID does not exist"
Sep 11 20:13:10.512726 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.512696    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr"]
Sep 11 20:13:10.513100 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.513034    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="435ff654-1f88-489e-90df-a18c422e98fe" containerName="test-curl-7565-a20e7d"
Sep 11 20:13:10.513100 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.513045    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="435ff654-1f88-489e-90df-a18c422e98fe" containerName="test-curl-7565-a20e7d"
Sep 11 20:13:10.513100 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.513101    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="435ff654-1f88-489e-90df-a18c422e98fe" containerName="test-curl-7565-a20e7d"
Sep 11 20:13:10.517426 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.517410    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr"
Sep 11 20:13:10.520010 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.519986    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-46da77-dockercfg-zgfsx\""
Sep 11 20:13:10.520132 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.520015    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-46da77\""
Sep 11 20:13:10.527053 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.527033    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr"]
Sep 11 20:13:10.577000 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.576976    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9"]
Sep 11 20:13:10.580376 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.580359    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9"
Sep 11 20:13:10.591415 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.591394    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9"]
Sep 11 20:13:10.671312 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.671287    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf4cc\" (UniqueName: \"kubernetes.io/projected/eeb00a15-0884-4dee-9ff4-5a444d79a148-kube-api-access-kf4cc\") pod \"payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9\" (UID: \"eeb00a15-0884-4dee-9ff4-5a444d79a148\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9"
Sep 11 20:13:10.671445 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.671321    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs5xf\" (UniqueName: \"kubernetes.io/projected/abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf-kube-api-access-fs5xf\") pod \"payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr\" (UID: \"abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf\") " pod="openshift-ingress/payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr"
Sep 11 20:13:10.671445 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.671345    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf-plugins-config-volume\") pod \"payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr\" (UID: \"abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf\") " pod="openshift-ingress/payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr"
Sep 11 20:13:10.671445 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.671422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eeb00a15-0884-4dee-9ff4-5a444d79a148-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9\" (UID: \"eeb00a15-0884-4dee-9ff4-5a444d79a148\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9"
Sep 11 20:13:10.772262 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.772191    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf4cc\" (UniqueName: \"kubernetes.io/projected/eeb00a15-0884-4dee-9ff4-5a444d79a148-kube-api-access-kf4cc\") pod \"payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9\" (UID: \"eeb00a15-0884-4dee-9ff4-5a444d79a148\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9"
Sep 11 20:13:10.772262 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.772230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs5xf\" (UniqueName: \"kubernetes.io/projected/abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf-kube-api-access-fs5xf\") pod \"payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr\" (UID: \"abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf\") " pod="openshift-ingress/payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr"
Sep 11 20:13:10.772262 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.772259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf-plugins-config-volume\") pod \"payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr\" (UID: \"abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf\") " pod="openshift-ingress/payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr"
Sep 11 20:13:10.772519 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.772324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eeb00a15-0884-4dee-9ff4-5a444d79a148-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9\" (UID: \"eeb00a15-0884-4dee-9ff4-5a444d79a148\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9"
Sep 11 20:13:10.773033 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.773012    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf-plugins-config-volume\") pod \"payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr\" (UID: \"abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf\") " pod="openshift-ingress/payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr"
Sep 11 20:13:10.773077 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.773047    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eeb00a15-0884-4dee-9ff4-5a444d79a148-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9\" (UID: \"eeb00a15-0884-4dee-9ff4-5a444d79a148\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9"
Sep 11 20:13:10.780781 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.780750    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf4cc\" (UniqueName: \"kubernetes.io/projected/eeb00a15-0884-4dee-9ff4-5a444d79a148-kube-api-access-kf4cc\") pod \"payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9\" (UID: \"eeb00a15-0884-4dee-9ff4-5a444d79a148\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9"
Sep 11 20:13:10.781176 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.781160    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs5xf\" (UniqueName: \"kubernetes.io/projected/abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf-kube-api-access-fs5xf\") pod \"payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr\" (UID: \"abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf\") " pod="openshift-ingress/payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr"
Sep 11 20:13:10.828468 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.828441    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr"
Sep 11 20:13:10.890895 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:10.890860    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9"
Sep 11 20:13:11.120599 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:11.120577    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr"]
Sep 11 20:13:11.122460 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:13:11.122435    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabcbf5fa_adc3_4cdf_b0e6_3b139a3ae9cf.slice/crio-120f799f00a6a0a5b65eb960453f37e595cdbc56d8400953a69346d2022c88f2 WatchSource:0}: Error finding container 120f799f00a6a0a5b65eb960453f37e595cdbc56d8400953a69346d2022c88f2: Status 404 returned error can't find the container with id 120f799f00a6a0a5b65eb960453f37e595cdbc56d8400953a69346d2022c88f2
Sep 11 20:13:11.214692 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:11.214667    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9"]
Sep 11 20:13:11.216864 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:13:11.216831    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb00a15_0884_4dee_9ff4_5a444d79a148.slice/crio-99ab31a9d13b3b9a9dc0a0104675c121511e3f2d82963bdb9e6f606afec6f949 WatchSource:0}: Error finding container 99ab31a9d13b3b9a9dc0a0104675c121511e3f2d82963bdb9e6f606afec6f949: Status 404 returned error can't find the container with id 99ab31a9d13b3b9a9dc0a0104675c121511e3f2d82963bdb9e6f606afec6f949
Sep 11 20:13:11.747955 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:11.747919    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9"]
Sep 11 20:13:11.752686 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:11.752656    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9"
Sep 11 20:13:11.755447 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:11.755406    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8e5fddd7\""
Sep 11 20:13:11.755591 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:11.755532    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8e5fddd7-dockercfg-4rdgt\""
Sep 11 20:13:11.764757 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:11.764735    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9"]
Sep 11 20:13:11.884659 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:11.884616    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a14e2397-bdfc-4fec-bf33-6c7c5782d044-plugins-config-volume\") pod \"payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9\" (UID: \"a14e2397-bdfc-4fec-bf33-6c7c5782d044\") " pod="openshift-ingress/payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9"
Sep 11 20:13:11.884888 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:11.884683    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbch6\" (UniqueName: \"kubernetes.io/projected/a14e2397-bdfc-4fec-bf33-6c7c5782d044-kube-api-access-nbch6\") pod \"payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9\" (UID: \"a14e2397-bdfc-4fec-bf33-6c7c5782d044\") " pod="openshift-ingress/payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9"
Sep 11 20:13:11.960155 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:11.960124    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx"
Sep 11 20:13:11.986113 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:11.986075    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a14e2397-bdfc-4fec-bf33-6c7c5782d044-plugins-config-volume\") pod \"payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9\" (UID: \"a14e2397-bdfc-4fec-bf33-6c7c5782d044\") " pod="openshift-ingress/payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9"
Sep 11 20:13:11.986277 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:11.986140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbch6\" (UniqueName: \"kubernetes.io/projected/a14e2397-bdfc-4fec-bf33-6c7c5782d044-kube-api-access-nbch6\") pod \"payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9\" (UID: \"a14e2397-bdfc-4fec-bf33-6c7c5782d044\") " pod="openshift-ingress/payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9"
Sep 11 20:13:11.986934 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:11.986905    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a14e2397-bdfc-4fec-bf33-6c7c5782d044-plugins-config-volume\") pod \"payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9\" (UID: \"a14e2397-bdfc-4fec-bf33-6c7c5782d044\") " pod="openshift-ingress/payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9"
Sep 11 20:13:11.994333 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:11.994304    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbch6\" (UniqueName: \"kubernetes.io/projected/a14e2397-bdfc-4fec-bf33-6c7c5782d044-kube-api-access-nbch6\") pod \"payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9\" (UID: \"a14e2397-bdfc-4fec-bf33-6c7c5782d044\") " pod="openshift-ingress/payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9"
Sep 11 20:13:12.026623 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:12.026531    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9" event={"ID":"eeb00a15-0884-4dee-9ff4-5a444d79a148","Type":"ContainerStarted","Data":"3054b8b256905d00ebed159fd0f24e603a3efcf4e358d204d82f4aed3c0967dc"}
Sep 11 20:13:12.026623 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:12.026587    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9" event={"ID":"eeb00a15-0884-4dee-9ff4-5a444d79a148","Type":"ContainerStarted","Data":"99ab31a9d13b3b9a9dc0a0104675c121511e3f2d82963bdb9e6f606afec6f949"}
Sep 11 20:13:12.026830 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:12.026636    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9"
Sep 11 20:13:12.028377 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:12.028353    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr" event={"ID":"abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf","Type":"ContainerStarted","Data":"e784143745d2a6574083b0f16207807a96759b8373f4f3950eca31d03da5e96a"}
Sep 11 20:13:12.028502 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:12.028396    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr"
Sep 11 20:13:12.028502 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:12.028412    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr" event={"ID":"abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf","Type":"ContainerStarted","Data":"120f799f00a6a0a5b65eb960453f37e595cdbc56d8400953a69346d2022c88f2"}
Sep 11 20:13:12.044791 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:12.044744    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9" podStartSLOduration=2.044729104 podStartE2EDuration="2.044729104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:12.042981926 +0000 UTC m=+762.035144207" watchObservedRunningTime="2026-09-11 20:13:12.044729104 +0000 UTC m=+762.036891400"
Sep 11 20:13:12.061721 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:12.061677    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr" podStartSLOduration=2.061663737 podStartE2EDuration="2.061663737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:12.059088496 +0000 UTC m=+762.051250768" watchObservedRunningTime="2026-09-11 20:13:12.061663737 +0000 UTC m=+762.053826030"
Sep 11 20:13:12.065434 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:12.065403    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9"
Sep 11 20:13:12.344380 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:12.344356    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9"]
Sep 11 20:13:12.346400 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:13:12.346370    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda14e2397_bdfc_4fec_bf33_6c7c5782d044.slice/crio-eb7048dd036ea86bc42d1dcd785a5817677298bbdbf7733a862ccd73eea42cf9 WatchSource:0}: Error finding container eb7048dd036ea86bc42d1dcd785a5817677298bbdbf7733a862ccd73eea42cf9: Status 404 returned error can't find the container with id eb7048dd036ea86bc42d1dcd785a5817677298bbdbf7733a862ccd73eea42cf9
Sep 11 20:13:12.970294 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:12.970260    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg"
Sep 11 20:13:12.970801 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:12.970780    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl"
Sep 11 20:13:13.036132 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:13.036096    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9" event={"ID":"a14e2397-bdfc-4fec-bf33-6c7c5782d044","Type":"ContainerStarted","Data":"06f06221f802302a4255097041c9acde2d7f3ea4350db2e5a5c791aed35688cd"}
Sep 11 20:13:13.036319 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:13.036150    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9" event={"ID":"a14e2397-bdfc-4fec-bf33-6c7c5782d044","Type":"ContainerStarted","Data":"eb7048dd036ea86bc42d1dcd785a5817677298bbdbf7733a862ccd73eea42cf9"}
Sep 11 20:13:13.036319 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:13.036197    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9"
Sep 11 20:13:13.118306 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:13.118253    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9" podStartSLOduration=2.118237791 podStartE2EDuration="2.118237791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:13.115974694 +0000 UTC m=+763.108136965" watchObservedRunningTime="2026-09-11 20:13:13.118237791 +0000 UTC m=+763.110400061"
Sep 11 20:13:13.976692 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:13.976662    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt"
Sep 11 20:13:14.984862 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:14.984831    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8"
Sep 11 20:13:18.329000 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:18.328960    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:13:22.231183 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:22.231149    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl"]
Sep 11 20:13:22.235716 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:22.235690    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl"
Sep 11 20:13:22.238667 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:22.238641    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4f9558-dockercfg-2hhbj\""
Sep 11 20:13:22.238807 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:22.238648    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4f9558\""
Sep 11 20:13:22.244747 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:22.244722    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl"]
Sep 11 20:13:22.380094 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:22.380062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp7c9\" (UniqueName: \"kubernetes.io/projected/6e079451-d9ce-46f4-ae94-012e2b81242c-kube-api-access-cp7c9\") pod \"payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl\" (UID: \"6e079451-d9ce-46f4-ae94-012e2b81242c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl"
Sep 11 20:13:22.380327 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:22.380164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e079451-d9ce-46f4-ae94-012e2b81242c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl\" (UID: \"6e079451-d9ce-46f4-ae94-012e2b81242c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl"
Sep 11 20:13:22.481372 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:22.481291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e079451-d9ce-46f4-ae94-012e2b81242c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl\" (UID: \"6e079451-d9ce-46f4-ae94-012e2b81242c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl"
Sep 11 20:13:22.481372 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:22.481344    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp7c9\" (UniqueName: \"kubernetes.io/projected/6e079451-d9ce-46f4-ae94-012e2b81242c-kube-api-access-cp7c9\") pod \"payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl\" (UID: \"6e079451-d9ce-46f4-ae94-012e2b81242c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl"
Sep 11 20:13:22.481957 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:22.481938    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e079451-d9ce-46f4-ae94-012e2b81242c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl\" (UID: \"6e079451-d9ce-46f4-ae94-012e2b81242c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl"
Sep 11 20:13:22.490532 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:22.490507    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp7c9\" (UniqueName: \"kubernetes.io/projected/6e079451-d9ce-46f4-ae94-012e2b81242c-kube-api-access-cp7c9\") pod \"payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl\" (UID: \"6e079451-d9ce-46f4-ae94-012e2b81242c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl"
Sep 11 20:13:22.547549 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:22.547525    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl"
Sep 11 20:13:22.822688 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:22.822662    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl"]
Sep 11 20:13:22.824680 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:13:22.824650    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e079451_d9ce_46f4_ae94_012e2b81242c.slice/crio-f0654209f969eff4b4c15e16c2580955de5132d7e22ee9b9c0e0ea8dd9f87ef5 WatchSource:0}: Error finding container f0654209f969eff4b4c15e16c2580955de5132d7e22ee9b9c0e0ea8dd9f87ef5: Status 404 returned error can't find the container with id f0654209f969eff4b4c15e16c2580955de5132d7e22ee9b9c0e0ea8dd9f87ef5
Sep 11 20:13:23.038965 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:23.038933    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr"
Sep 11 20:13:23.039101 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:23.039092    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9"
Sep 11 20:13:23.096090 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:23.095985    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl" event={"ID":"6e079451-d9ce-46f4-ae94-012e2b81242c","Type":"ContainerStarted","Data":"f2aba01ab42a8ced212cd0f834a1d4a4519921823b757c017df62fb9020775d0"}
Sep 11 20:13:23.096090 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:23.096054    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl" event={"ID":"6e079451-d9ce-46f4-ae94-012e2b81242c","Type":"ContainerStarted","Data":"f0654209f969eff4b4c15e16c2580955de5132d7e22ee9b9c0e0ea8dd9f87ef5"}
Sep 11 20:13:23.096310 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:23.096117    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl"
Sep 11 20:13:23.122834 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:23.115107    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl" podStartSLOduration=1.115088202 podStartE2EDuration="1.115088202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:23.114671787 +0000 UTC m=+773.106834072" watchObservedRunningTime="2026-09-11 20:13:23.115088202 +0000 UTC m=+773.107250473"
Sep 11 20:13:24.045229 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:24.045197    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9"
Sep 11 20:13:26.487612 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:26.487581    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e5fddd7-4drq2"]
Sep 11 20:13:26.491204 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:26.491187    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e5fddd7-4drq2"
Sep 11 20:13:26.493827 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:26.493801    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:13:26.493941 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:26.493892    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7dbs5\""
Sep 11 20:13:26.499766 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:26.499741    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e5fddd7-4drq2"]
Sep 11 20:13:26.614744 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:26.614710    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67bb7cd0-7063-44b9-9110-346d66d9156a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8e5fddd7-4drq2\" (UID: \"67bb7cd0-7063-44b9-9110-346d66d9156a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e5fddd7-4drq2"
Sep 11 20:13:26.715936 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:26.715901    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67bb7cd0-7063-44b9-9110-346d66d9156a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8e5fddd7-4drq2\" (UID: \"67bb7cd0-7063-44b9-9110-346d66d9156a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e5fddd7-4drq2"
Sep 11 20:13:26.716554 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:26.716533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67bb7cd0-7063-44b9-9110-346d66d9156a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8e5fddd7-4drq2\" (UID: \"67bb7cd0-7063-44b9-9110-346d66d9156a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e5fddd7-4drq2"
Sep 11 20:13:26.802977 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:26.802905    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e5fddd7-4drq2"
Sep 11 20:13:27.079180 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:27.079154    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e5fddd7-4drq2"]
Sep 11 20:13:27.081132 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:13:27.081104    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67bb7cd0_7063_44b9_9110_346d66d9156a.slice/crio-2ce4d7ae1bf5cc16e950ad46b8c07dbfbde2a0ef4a5020d2170942c2fdbce53a WatchSource:0}: Error finding container 2ce4d7ae1bf5cc16e950ad46b8c07dbfbde2a0ef4a5020d2170942c2fdbce53a: Status 404 returned error can't find the container with id 2ce4d7ae1bf5cc16e950ad46b8c07dbfbde2a0ef4a5020d2170942c2fdbce53a
Sep 11 20:13:27.120444 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:27.120413    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e5fddd7-4drq2" event={"ID":"67bb7cd0-7063-44b9-9110-346d66d9156a","Type":"ContainerStarted","Data":"2ce4d7ae1bf5cc16e950ad46b8c07dbfbde2a0ef4a5020d2170942c2fdbce53a"}
Sep 11 20:13:30.142475 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:30.142375    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e5fddd7-4drq2" event={"ID":"67bb7cd0-7063-44b9-9110-346d66d9156a","Type":"ContainerStarted","Data":"5602f117668c0ba1a0cd9998aafdd8aa58c5ae34a7fc7d935f70ffca8ac79235"}
Sep 11 20:13:30.166122 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:30.166073    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e5fddd7-4drq2" podStartSLOduration=1.707617168 podStartE2EDuration="4.166055948s" totalImagesPullingTime="2.45843878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:30.163373439 +0000 UTC m=+780.155535714" watchObservedRunningTime="2026-09-11 20:13:30.166055948 +0000 UTC m=+780.158218254"
Sep 11 20:13:31.149391 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:31.149357    2580 generic.go:356] "Generic (PLEG): container finished" podID="67bb7cd0-7063-44b9-9110-346d66d9156a" containerID="5602f117668c0ba1a0cd9998aafdd8aa58c5ae34a7fc7d935f70ffca8ac79235" exitCode=0
Sep 11 20:13:31.149802 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:31.149447    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e5fddd7-4drq2" event={"ID":"67bb7cd0-7063-44b9-9110-346d66d9156a","Type":"ContainerDied","Data":"5602f117668c0ba1a0cd9998aafdd8aa58c5ae34a7fc7d935f70ffca8ac79235"}
Sep 11 20:13:32.283817 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:32.283791    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e5fddd7-4drq2"
Sep 11 20:13:32.366948 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:32.366918    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67bb7cd0-7063-44b9-9110-346d66d9156a-maas-api-service-ca\") pod \"67bb7cd0-7063-44b9-9110-346d66d9156a\" (UID: \"67bb7cd0-7063-44b9-9110-346d66d9156a\") "
Sep 11 20:13:32.367281 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:32.367257    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67bb7cd0-7063-44b9-9110-346d66d9156a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "67bb7cd0-7063-44b9-9110-346d66d9156a" (UID: "67bb7cd0-7063-44b9-9110-346d66d9156a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:13:32.468437 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:32.468405    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67bb7cd0-7063-44b9-9110-346d66d9156a-maas-api-service-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:13:33.167151 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:33.167120    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e5fddd7-4drq2"
Sep 11 20:13:33.167323 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:33.167149    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e5fddd7-4drq2" event={"ID":"67bb7cd0-7063-44b9-9110-346d66d9156a","Type":"ContainerDied","Data":"2ce4d7ae1bf5cc16e950ad46b8c07dbfbde2a0ef4a5020d2170942c2fdbce53a"}
Sep 11 20:13:33.167323 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:33.167214    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ce4d7ae1bf5cc16e950ad46b8c07dbfbde2a0ef4a5020d2170942c2fdbce53a"
Sep 11 20:13:34.104353 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:34.104315    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl"
Sep 11 20:13:37.073106 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:37.073074    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9"]
Sep 11 20:13:37.073586 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:37.073323    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9" podUID="a14e2397-bdfc-4fec-bf33-6c7c5782d044" containerName="payload-processing" containerID="cri-o://06f06221f802302a4255097041c9acde2d7f3ea4350db2e5a5c791aed35688cd" gracePeriod=30
Sep 11 20:13:37.191272 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:37.191244    2580 generic.go:356] "Generic (PLEG): container finished" podID="a14e2397-bdfc-4fec-bf33-6c7c5782d044" containerID="06f06221f802302a4255097041c9acde2d7f3ea4350db2e5a5c791aed35688cd" exitCode=0
Sep 11 20:13:37.191384 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:37.191317    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9" event={"ID":"a14e2397-bdfc-4fec-bf33-6c7c5782d044","Type":"ContainerDied","Data":"06f06221f802302a4255097041c9acde2d7f3ea4350db2e5a5c791aed35688cd"}
Sep 11 20:13:37.241327 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:37.241266    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e5fddd7-4drq2"]
Sep 11 20:13:37.244109 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:37.244085    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8e5fddd7-4drq2"]
Sep 11 20:13:37.332718 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:37.332655    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9"
Sep 11 20:13:37.409806 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:37.409780    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a14e2397-bdfc-4fec-bf33-6c7c5782d044-plugins-config-volume\") pod \"a14e2397-bdfc-4fec-bf33-6c7c5782d044\" (UID: \"a14e2397-bdfc-4fec-bf33-6c7c5782d044\") "
Sep 11 20:13:37.409986 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:37.409824    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbch6\" (UniqueName: \"kubernetes.io/projected/a14e2397-bdfc-4fec-bf33-6c7c5782d044-kube-api-access-nbch6\") pod \"a14e2397-bdfc-4fec-bf33-6c7c5782d044\" (UID: \"a14e2397-bdfc-4fec-bf33-6c7c5782d044\") "
Sep 11 20:13:37.410132 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:37.410109    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a14e2397-bdfc-4fec-bf33-6c7c5782d044-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a14e2397-bdfc-4fec-bf33-6c7c5782d044" (UID: "a14e2397-bdfc-4fec-bf33-6c7c5782d044"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:13:37.411805 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:37.411782    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a14e2397-bdfc-4fec-bf33-6c7c5782d044-kube-api-access-nbch6" (OuterVolumeSpecName: "kube-api-access-nbch6") pod "a14e2397-bdfc-4fec-bf33-6c7c5782d044" (UID: "a14e2397-bdfc-4fec-bf33-6c7c5782d044"). InnerVolumeSpecName "kube-api-access-nbch6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:13:37.510364 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:37.510332    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbch6\" (UniqueName: \"kubernetes.io/projected/a14e2397-bdfc-4fec-bf33-6c7c5782d044-kube-api-access-nbch6\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:13:37.510364 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:37.510361    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a14e2397-bdfc-4fec-bf33-6c7c5782d044-plugins-config-volume\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:13:38.197626 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:38.197597    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9"
Sep 11 20:13:38.197626 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:38.197610    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9" event={"ID":"a14e2397-bdfc-4fec-bf33-6c7c5782d044","Type":"ContainerDied","Data":"eb7048dd036ea86bc42d1dcd785a5817677298bbdbf7733a862ccd73eea42cf9"}
Sep 11 20:13:38.198137 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:38.197666    2580 scope.go:117] "RemoveContainer" containerID="06f06221f802302a4255097041c9acde2d7f3ea4350db2e5a5c791aed35688cd"
Sep 11 20:13:38.220084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:38.220060    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9"]
Sep 11 20:13:38.222346 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:38.222326    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8e5fddd7-55b6b5ccc-td5r9"]
Sep 11 20:13:38.441662 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:38.441629    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67bb7cd0-7063-44b9-9110-346d66d9156a" path="/var/lib/kubelet/pods/67bb7cd0-7063-44b9-9110-346d66d9156a/volumes"
Sep 11 20:13:38.441996 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:38.441982    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a14e2397-bdfc-4fec-bf33-6c7c5782d044" path="/var/lib/kubelet/pods/a14e2397-bdfc-4fec-bf33-6c7c5782d044/volumes"
Sep 11 20:13:43.812293 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:43.812257    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd"]
Sep 11 20:13:43.812749 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:43.812737    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a14e2397-bdfc-4fec-bf33-6c7c5782d044" containerName="payload-processing"
Sep 11 20:13:43.812797 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:43.812753    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a14e2397-bdfc-4fec-bf33-6c7c5782d044" containerName="payload-processing"
Sep 11 20:13:43.812797 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:43.812765    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67bb7cd0-7063-44b9-9110-346d66d9156a" containerName="revoke-keys"
Sep 11 20:13:43.812797 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:43.812771    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="67bb7cd0-7063-44b9-9110-346d66d9156a" containerName="revoke-keys"
Sep 11 20:13:43.812889 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:43.812840    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a14e2397-bdfc-4fec-bf33-6c7c5782d044" containerName="payload-processing"
Sep 11 20:13:43.812889 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:43.812850    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="67bb7cd0-7063-44b9-9110-346d66d9156a" containerName="revoke-keys"
Sep 11 20:13:43.815626 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:43.815610    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd"
Sep 11 20:13:43.818280 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:43.818258    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-c83b7433\""
Sep 11 20:13:43.818470 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:43.818455    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-c83b7433-dockercfg-25mmr\""
Sep 11 20:13:43.833445 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:43.833415    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd"]
Sep 11 20:13:43.864312 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:43.864284    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88l6c\" (UniqueName: \"kubernetes.io/projected/e917f438-94a5-4e3a-8aaf-14a239729ecf-kube-api-access-88l6c\") pod \"payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd\" (UID: \"e917f438-94a5-4e3a-8aaf-14a239729ecf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd"
Sep 11 20:13:43.864436 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:43.864340    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e917f438-94a5-4e3a-8aaf-14a239729ecf-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd\" (UID: \"e917f438-94a5-4e3a-8aaf-14a239729ecf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd"
Sep 11 20:13:43.965804 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:43.965773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e917f438-94a5-4e3a-8aaf-14a239729ecf-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd\" (UID: \"e917f438-94a5-4e3a-8aaf-14a239729ecf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd"
Sep 11 20:13:43.965933 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:43.965842    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88l6c\" (UniqueName: \"kubernetes.io/projected/e917f438-94a5-4e3a-8aaf-14a239729ecf-kube-api-access-88l6c\") pod \"payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd\" (UID: \"e917f438-94a5-4e3a-8aaf-14a239729ecf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd"
Sep 11 20:13:43.966539 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:43.966513    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e917f438-94a5-4e3a-8aaf-14a239729ecf-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd\" (UID: \"e917f438-94a5-4e3a-8aaf-14a239729ecf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd"
Sep 11 20:13:43.973875 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:43.973845    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88l6c\" (UniqueName: \"kubernetes.io/projected/e917f438-94a5-4e3a-8aaf-14a239729ecf-kube-api-access-88l6c\") pod \"payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd\" (UID: \"e917f438-94a5-4e3a-8aaf-14a239729ecf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd"
Sep 11 20:13:44.126992 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:44.126906    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd"
Sep 11 20:13:44.398285 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:44.398252    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd"]
Sep 11 20:13:44.399843 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:13:44.399815    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode917f438_94a5_4e3a_8aaf_14a239729ecf.slice/crio-6e90cc6c6745d91f42b5d560269be519c714cd598f4adf9ab8b0e665a17fef76 WatchSource:0}: Error finding container 6e90cc6c6745d91f42b5d560269be519c714cd598f4adf9ab8b0e665a17fef76: Status 404 returned error can't find the container with id 6e90cc6c6745d91f42b5d560269be519c714cd598f4adf9ab8b0e665a17fef76
Sep 11 20:13:45.238510 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:45.238461    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd" event={"ID":"e917f438-94a5-4e3a-8aaf-14a239729ecf","Type":"ContainerStarted","Data":"a70a67823993b5d56abb4cf0c9800317d0983283b3409427a15d41bc838129ea"}
Sep 11 20:13:45.238883 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:45.238574    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd" event={"ID":"e917f438-94a5-4e3a-8aaf-14a239729ecf","Type":"ContainerStarted","Data":"6e90cc6c6745d91f42b5d560269be519c714cd598f4adf9ab8b0e665a17fef76"}
Sep 11 20:13:45.238883 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:45.238605    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd"
Sep 11 20:13:45.256467 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:45.256422    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd" podStartSLOduration=2.25640884 podStartE2EDuration="2.25640884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:45.254837358 +0000 UTC m=+795.246999629" watchObservedRunningTime="2026-09-11 20:13:45.25640884 +0000 UTC m=+795.248571124"
Sep 11 20:13:48.616873 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:48.616826    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:13:56.246396 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:13:56.246316    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd"
Sep 11 20:14:05.702305 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:05.702269    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c83b7433-cz7l6"]
Sep 11 20:14:05.710058 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:05.710040    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c83b7433-cz7l6"
Sep 11 20:14:05.713017 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:05.712999    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7dbs5\""
Sep 11 20:14:05.713123 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:05.713002    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:14:05.723149 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:05.723124    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c83b7433-cz7l6"]
Sep 11 20:14:05.854103 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:05.854063    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd74d86-ed43-4b16-ad72-ba8cf24660fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c83b7433-cz7l6\" (UID: \"5bd74d86-ed43-4b16-ad72-ba8cf24660fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c83b7433-cz7l6"
Sep 11 20:14:05.954851 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:05.954775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd74d86-ed43-4b16-ad72-ba8cf24660fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c83b7433-cz7l6\" (UID: \"5bd74d86-ed43-4b16-ad72-ba8cf24660fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c83b7433-cz7l6"
Sep 11 20:14:05.955407 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:05.955381    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd74d86-ed43-4b16-ad72-ba8cf24660fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c83b7433-cz7l6\" (UID: \"5bd74d86-ed43-4b16-ad72-ba8cf24660fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c83b7433-cz7l6"
Sep 11 20:14:06.020576 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:06.020529    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c83b7433-cz7l6"
Sep 11 20:14:06.291711 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:06.291683    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c83b7433-cz7l6"]
Sep 11 20:14:06.293440 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:14:06.293407    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd74d86_ed43_4b16_ad72_ba8cf24660fe.slice/crio-6224c6cb5b5fa678a08cb889ae73eeee38270cbfb8745298426b7ab37530d201 WatchSource:0}: Error finding container 6224c6cb5b5fa678a08cb889ae73eeee38270cbfb8745298426b7ab37530d201: Status 404 returned error can't find the container with id 6224c6cb5b5fa678a08cb889ae73eeee38270cbfb8745298426b7ab37530d201
Sep 11 20:14:06.354942 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:06.354915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c83b7433-cz7l6" event={"ID":"5bd74d86-ed43-4b16-ad72-ba8cf24660fe","Type":"ContainerStarted","Data":"6224c6cb5b5fa678a08cb889ae73eeee38270cbfb8745298426b7ab37530d201"}
Sep 11 20:14:07.362303 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:07.362209    2580 generic.go:356] "Generic (PLEG): container finished" podID="5bd74d86-ed43-4b16-ad72-ba8cf24660fe" containerID="39df5188ec8960b0f6e73dbbfc9108bbde89b6a62fe6fc9c13887739e15d5c02" exitCode=0
Sep 11 20:14:07.362725 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:07.362301    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c83b7433-cz7l6" event={"ID":"5bd74d86-ed43-4b16-ad72-ba8cf24660fe","Type":"ContainerDied","Data":"39df5188ec8960b0f6e73dbbfc9108bbde89b6a62fe6fc9c13887739e15d5c02"}
Sep 11 20:14:08.515006 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:08.514981    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c83b7433-cz7l6"
Sep 11 20:14:08.677677 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:08.677574    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd74d86-ed43-4b16-ad72-ba8cf24660fe-maas-api-service-ca\") pod \"5bd74d86-ed43-4b16-ad72-ba8cf24660fe\" (UID: \"5bd74d86-ed43-4b16-ad72-ba8cf24660fe\") "
Sep 11 20:14:08.677948 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:08.677923    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bd74d86-ed43-4b16-ad72-ba8cf24660fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5bd74d86-ed43-4b16-ad72-ba8cf24660fe" (UID: "5bd74d86-ed43-4b16-ad72-ba8cf24660fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:14:08.779204 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:08.779165    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd74d86-ed43-4b16-ad72-ba8cf24660fe-maas-api-service-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:14:09.374256 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:09.374228    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c83b7433-cz7l6"
Sep 11 20:14:09.374425 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:09.374252    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c83b7433-cz7l6" event={"ID":"5bd74d86-ed43-4b16-ad72-ba8cf24660fe","Type":"ContainerDied","Data":"6224c6cb5b5fa678a08cb889ae73eeee38270cbfb8745298426b7ab37530d201"}
Sep 11 20:14:09.374425 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:09.374319    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6224c6cb5b5fa678a08cb889ae73eeee38270cbfb8745298426b7ab37530d201"
Sep 11 20:14:11.211520 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:11.211471    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:14:12.242796 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:12.242763    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd"]
Sep 11 20:14:12.243375 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:12.243027    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd" podUID="e917f438-94a5-4e3a-8aaf-14a239729ecf" containerName="payload-pre-processing" containerID="cri-o://a70a67823993b5d56abb4cf0c9800317d0983283b3409427a15d41bc838129ea" gracePeriod=30
Sep 11 20:14:12.396002 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:12.395961    2580 generic.go:356] "Generic (PLEG): container finished" podID="e917f438-94a5-4e3a-8aaf-14a239729ecf" containerID="a70a67823993b5d56abb4cf0c9800317d0983283b3409427a15d41bc838129ea" exitCode=0
Sep 11 20:14:12.396211 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:12.396040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd" event={"ID":"e917f438-94a5-4e3a-8aaf-14a239729ecf","Type":"ContainerDied","Data":"a70a67823993b5d56abb4cf0c9800317d0983283b3409427a15d41bc838129ea"}
Sep 11 20:14:12.421281 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:12.421249    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c83b7433-cz7l6"]
Sep 11 20:14:12.425175 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:12.425146    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c83b7433-cz7l6"]
Sep 11 20:14:12.443282 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:12.443249    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bd74d86-ed43-4b16-ad72-ba8cf24660fe" path="/var/lib/kubelet/pods/5bd74d86-ed43-4b16-ad72-ba8cf24660fe/volumes"
Sep 11 20:14:12.530664 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:12.530637    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd"
Sep 11 20:14:12.717713 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:12.717670    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88l6c\" (UniqueName: \"kubernetes.io/projected/e917f438-94a5-4e3a-8aaf-14a239729ecf-kube-api-access-88l6c\") pod \"e917f438-94a5-4e3a-8aaf-14a239729ecf\" (UID: \"e917f438-94a5-4e3a-8aaf-14a239729ecf\") "
Sep 11 20:14:12.717908 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:12.717759    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e917f438-94a5-4e3a-8aaf-14a239729ecf-plugins-config-volume\") pod \"e917f438-94a5-4e3a-8aaf-14a239729ecf\" (UID: \"e917f438-94a5-4e3a-8aaf-14a239729ecf\") "
Sep 11 20:14:12.718157 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:12.718130    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e917f438-94a5-4e3a-8aaf-14a239729ecf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e917f438-94a5-4e3a-8aaf-14a239729ecf" (UID: "e917f438-94a5-4e3a-8aaf-14a239729ecf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:14:12.719722 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:12.719700    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e917f438-94a5-4e3a-8aaf-14a239729ecf-kube-api-access-88l6c" (OuterVolumeSpecName: "kube-api-access-88l6c") pod "e917f438-94a5-4e3a-8aaf-14a239729ecf" (UID: "e917f438-94a5-4e3a-8aaf-14a239729ecf"). InnerVolumeSpecName "kube-api-access-88l6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:14:12.819330 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:12.819238    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88l6c\" (UniqueName: \"kubernetes.io/projected/e917f438-94a5-4e3a-8aaf-14a239729ecf-kube-api-access-88l6c\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:14:12.819330 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:12.819272    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e917f438-94a5-4e3a-8aaf-14a239729ecf-plugins-config-volume\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:14:13.403377 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:13.403341    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd"
Sep 11 20:14:13.403377 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:13.403354    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd" event={"ID":"e917f438-94a5-4e3a-8aaf-14a239729ecf","Type":"ContainerDied","Data":"6e90cc6c6745d91f42b5d560269be519c714cd598f4adf9ab8b0e665a17fef76"}
Sep 11 20:14:13.403873 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:13.403450    2580 scope.go:117] "RemoveContainer" containerID="a70a67823993b5d56abb4cf0c9800317d0983283b3409427a15d41bc838129ea"
Sep 11 20:14:13.426046 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:13.426019    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd"]
Sep 11 20:14:13.431566 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:13.431544    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c83b7433-6b94954786-scrmd"]
Sep 11 20:14:14.440972 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:14.440938    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e917f438-94a5-4e3a-8aaf-14a239729ecf" path="/var/lib/kubelet/pods/e917f438-94a5-4e3a-8aaf-14a239729ecf/volumes"
Sep 11 20:14:34.818564 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:34.818511    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:14:38.946924 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:38.946890    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"]
Sep 11 20:14:38.947417 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:38.947254    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e917f438-94a5-4e3a-8aaf-14a239729ecf" containerName="payload-pre-processing"
Sep 11 20:14:38.947417 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:38.947265    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e917f438-94a5-4e3a-8aaf-14a239729ecf" containerName="payload-pre-processing"
Sep 11 20:14:38.947417 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:38.947286    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bd74d86-ed43-4b16-ad72-ba8cf24660fe" containerName="revoke-keys"
Sep 11 20:14:38.947417 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:38.947292    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bd74d86-ed43-4b16-ad72-ba8cf24660fe" containerName="revoke-keys"
Sep 11 20:14:38.947417 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:38.947349    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bd74d86-ed43-4b16-ad72-ba8cf24660fe" containerName="revoke-keys"
Sep 11 20:14:38.947417 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:38.947358    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e917f438-94a5-4e3a-8aaf-14a239729ecf" containerName="payload-pre-processing"
Sep 11 20:14:38.951581 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:38.951564    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:14:38.954879 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:38.954851    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9d9abb\"/\"default-dockercfg-56qbm\""
Sep 11 20:14:38.956122 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:38.956101    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9d9abb\"/\"simulator-w0-9d9abb-kserve-self-signed-certs\""
Sep 11 20:14:38.956248 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:38.956108    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9d9abb\"/\"kube-root-ca.crt\""
Sep 11 20:14:38.956797 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:38.956779    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9d9abb\"/\"openshift-service-ca.crt\""
Sep 11 20:14:38.971550 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:38.971526    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"]
Sep 11 20:14:39.047924 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:39.047893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/586ad6db-0f64-4427-8e12-6e13856bea86-home\") pod \"simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn\" (UID: \"586ad6db-0f64-4427-8e12-6e13856bea86\") " pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:14:39.048078 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:39.047940    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/586ad6db-0f64-4427-8e12-6e13856bea86-model-cache\") pod \"simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn\" (UID: \"586ad6db-0f64-4427-8e12-6e13856bea86\") " pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:14:39.048078 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:39.048022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/586ad6db-0f64-4427-8e12-6e13856bea86-dshm\") pod \"simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn\" (UID: \"586ad6db-0f64-4427-8e12-6e13856bea86\") " pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:14:39.048078 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:39.048045    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-479n2\" (UniqueName: \"kubernetes.io/projected/586ad6db-0f64-4427-8e12-6e13856bea86-kube-api-access-479n2\") pod \"simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn\" (UID: \"586ad6db-0f64-4427-8e12-6e13856bea86\") " pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:14:39.048196 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:39.048097    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/586ad6db-0f64-4427-8e12-6e13856bea86-tls-certs\") pod \"simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn\" (UID: \"586ad6db-0f64-4427-8e12-6e13856bea86\") " pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:14:39.048196 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:39.048162    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/586ad6db-0f64-4427-8e12-6e13856bea86-tmp-dir\") pod \"simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn\" (UID: \"586ad6db-0f64-4427-8e12-6e13856bea86\") " pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:14:39.149450 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:39.149413    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/586ad6db-0f64-4427-8e12-6e13856bea86-home\") pod \"simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn\" (UID: \"586ad6db-0f64-4427-8e12-6e13856bea86\") " pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:14:39.149450 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:39.149462    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/586ad6db-0f64-4427-8e12-6e13856bea86-model-cache\") pod \"simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn\" (UID: \"586ad6db-0f64-4427-8e12-6e13856bea86\") " pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:14:39.149720 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:39.149532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/586ad6db-0f64-4427-8e12-6e13856bea86-dshm\") pod \"simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn\" (UID: \"586ad6db-0f64-4427-8e12-6e13856bea86\") " pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:14:39.149720 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:39.149553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-479n2\" (UniqueName: \"kubernetes.io/projected/586ad6db-0f64-4427-8e12-6e13856bea86-kube-api-access-479n2\") pod \"simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn\" (UID: \"586ad6db-0f64-4427-8e12-6e13856bea86\") " pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:14:39.149720 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:39.149572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/586ad6db-0f64-4427-8e12-6e13856bea86-tls-certs\") pod \"simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn\" (UID: \"586ad6db-0f64-4427-8e12-6e13856bea86\") " pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:14:39.149720 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:39.149617    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/586ad6db-0f64-4427-8e12-6e13856bea86-tmp-dir\") pod \"simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn\" (UID: \"586ad6db-0f64-4427-8e12-6e13856bea86\") " pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:14:39.149958 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:39.149910    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/586ad6db-0f64-4427-8e12-6e13856bea86-home\") pod \"simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn\" (UID: \"586ad6db-0f64-4427-8e12-6e13856bea86\") " pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:14:39.149958 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:39.149925    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/586ad6db-0f64-4427-8e12-6e13856bea86-model-cache\") pod \"simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn\" (UID: \"586ad6db-0f64-4427-8e12-6e13856bea86\") " pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:14:39.150049 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:39.149995    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/586ad6db-0f64-4427-8e12-6e13856bea86-tmp-dir\") pod \"simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn\" (UID: \"586ad6db-0f64-4427-8e12-6e13856bea86\") " pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:14:39.151785 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:39.151760    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/586ad6db-0f64-4427-8e12-6e13856bea86-dshm\") pod \"simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn\" (UID: \"586ad6db-0f64-4427-8e12-6e13856bea86\") " pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:14:39.152071 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:39.152052    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/586ad6db-0f64-4427-8e12-6e13856bea86-tls-certs\") pod \"simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn\" (UID: \"586ad6db-0f64-4427-8e12-6e13856bea86\") " pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:14:39.157618 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:39.157596    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-479n2\" (UniqueName: \"kubernetes.io/projected/586ad6db-0f64-4427-8e12-6e13856bea86-kube-api-access-479n2\") pod \"simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn\" (UID: \"586ad6db-0f64-4427-8e12-6e13856bea86\") " pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:14:39.260845 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:39.260756    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:14:39.533020 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:39.532995    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"]
Sep 11 20:14:39.534901 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:14:39.534877    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod586ad6db_0f64_4427_8e12_6e13856bea86.slice/crio-60b5d66a2fdddbb9dcc765acba2955d2d0a25e6842f2ad9d34d6e3cd2c643af7 WatchSource:0}: Error finding container 60b5d66a2fdddbb9dcc765acba2955d2d0a25e6842f2ad9d34d6e3cd2c643af7: Status 404 returned error can't find the container with id 60b5d66a2fdddbb9dcc765acba2955d2d0a25e6842f2ad9d34d6e3cd2c643af7
Sep 11 20:14:39.543437 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:39.543410    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn" event={"ID":"586ad6db-0f64-4427-8e12-6e13856bea86","Type":"ContainerStarted","Data":"60b5d66a2fdddbb9dcc765acba2955d2d0a25e6842f2ad9d34d6e3cd2c643af7"}
Sep 11 20:14:40.551428 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:40.551390    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn" event={"ID":"586ad6db-0f64-4427-8e12-6e13856bea86","Type":"ContainerStarted","Data":"a2eed54ead3af62886d6b4bf6cb6b621ff3508bb0eb89988f04d190b38886e59"}
Sep 11 20:14:40.571553 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:40.571474    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn" podStartSLOduration=2.571458221 podStartE2EDuration="2.571458221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:14:40.569129297 +0000 UTC m=+850.561291569" watchObservedRunningTime="2026-09-11 20:14:40.571458221 +0000 UTC m=+850.563620491"
Sep 11 20:14:41.296829 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:41.296786    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cd58ce-pjfzn"]
Sep 11 20:14:41.300319 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:41.300295    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cd58ce-pjfzn"
Sep 11 20:14:41.303053 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:41.303032    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7dbs5\""
Sep 11 20:14:41.303161 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:41.303058    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:14:41.307698 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:41.307673    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cd58ce-pjfzn"]
Sep 11 20:14:41.369389 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:41.369355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ee0409-c96d-4bb6-a55a-4dd23666154c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-16cd58ce-pjfzn\" (UID: \"09ee0409-c96d-4bb6-a55a-4dd23666154c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cd58ce-pjfzn"
Sep 11 20:14:41.470899 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:41.470862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ee0409-c96d-4bb6-a55a-4dd23666154c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-16cd58ce-pjfzn\" (UID: \"09ee0409-c96d-4bb6-a55a-4dd23666154c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cd58ce-pjfzn"
Sep 11 20:14:41.471555 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:41.471532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ee0409-c96d-4bb6-a55a-4dd23666154c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-16cd58ce-pjfzn\" (UID: \"09ee0409-c96d-4bb6-a55a-4dd23666154c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cd58ce-pjfzn"
Sep 11 20:14:41.613228 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:41.613133    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cd58ce-pjfzn"
Sep 11 20:14:41.890784 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:41.889409    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cd58ce-pjfzn"]
Sep 11 20:14:41.896015 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:14:41.895982    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ee0409_c96d_4bb6_a55a_4dd23666154c.slice/crio-8fdf7f6346e4686020f618475269e2b5fe903603cc7cb154e26cb64e909e8b49 WatchSource:0}: Error finding container 8fdf7f6346e4686020f618475269e2b5fe903603cc7cb154e26cb64e909e8b49: Status 404 returned error can't find the container with id 8fdf7f6346e4686020f618475269e2b5fe903603cc7cb154e26cb64e909e8b49
Sep 11 20:14:42.563789 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:42.563747    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cd58ce-pjfzn" event={"ID":"09ee0409-c96d-4bb6-a55a-4dd23666154c","Type":"ContainerStarted","Data":"2568423964dae6f8bd6ae0ff15b7068359bd74e97cb2b4ff2d88b462fe30a256"}
Sep 11 20:14:42.563995 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:42.563803    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cd58ce-pjfzn" event={"ID":"09ee0409-c96d-4bb6-a55a-4dd23666154c","Type":"ContainerStarted","Data":"8fdf7f6346e4686020f618475269e2b5fe903603cc7cb154e26cb64e909e8b49"}
Sep 11 20:14:42.580672 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:42.580624    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cd58ce-pjfzn" podStartSLOduration=1.580609467 podStartE2EDuration="1.580609467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:14:42.57856318 +0000 UTC m=+852.570725449" watchObservedRunningTime="2026-09-11 20:14:42.580609467 +0000 UTC m=+852.572771736"
Sep 11 20:14:43.571107 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:43.571074    2580 generic.go:356] "Generic (PLEG): container finished" podID="09ee0409-c96d-4bb6-a55a-4dd23666154c" containerID="2568423964dae6f8bd6ae0ff15b7068359bd74e97cb2b4ff2d88b462fe30a256" exitCode=0
Sep 11 20:14:43.571517 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:43.571132    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cd58ce-pjfzn" event={"ID":"09ee0409-c96d-4bb6-a55a-4dd23666154c","Type":"ContainerDied","Data":"2568423964dae6f8bd6ae0ff15b7068359bd74e97cb2b4ff2d88b462fe30a256"}
Sep 11 20:14:44.585087 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.585049    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"]
Sep 11 20:14:44.588853 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.588833    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:14:44.593079 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.593055    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-46da77\"/\"kube-root-ca.crt\""
Sep 11 20:14:44.593246 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.593083    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-46da77\"/\"autores-model-46da77-kserve-self-signed-certs\""
Sep 11 20:14:44.593246 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.593083    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-46da77\"/\"default-dockercfg-lc64d\""
Sep 11 20:14:44.593246 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.593140    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-46da77\"/\"openshift-service-ca.crt\""
Sep 11 20:14:44.601273 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.601241    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"]
Sep 11 20:14:44.701579 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.701541    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-tls-certs\") pod \"autores-model-46da77-kserve-5f56fd66c-m8z9v\" (UID: \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\") " pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:14:44.701755 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.701582    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqfww\" (UniqueName: \"kubernetes.io/projected/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-kube-api-access-rqfww\") pod \"autores-model-46da77-kserve-5f56fd66c-m8z9v\" (UID: \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\") " pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:14:44.701755 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.701656    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-model-cache\") pod \"autores-model-46da77-kserve-5f56fd66c-m8z9v\" (UID: \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\") " pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:14:44.701755 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.701702    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-dshm\") pod \"autores-model-46da77-kserve-5f56fd66c-m8z9v\" (UID: \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\") " pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:14:44.701755 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.701727    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-tmp-dir\") pod \"autores-model-46da77-kserve-5f56fd66c-m8z9v\" (UID: \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\") " pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:14:44.701919 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.701807    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-home\") pod \"autores-model-46da77-kserve-5f56fd66c-m8z9v\" (UID: \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\") " pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:14:44.744192 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.744164    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cd58ce-pjfzn"
Sep 11 20:14:44.802541 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.802504    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-tmp-dir\") pod \"autores-model-46da77-kserve-5f56fd66c-m8z9v\" (UID: \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\") " pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:14:44.802731 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.802586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-home\") pod \"autores-model-46da77-kserve-5f56fd66c-m8z9v\" (UID: \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\") " pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:14:44.802731 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.802628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-tls-certs\") pod \"autores-model-46da77-kserve-5f56fd66c-m8z9v\" (UID: \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\") " pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:14:44.802731 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.802646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqfww\" (UniqueName: \"kubernetes.io/projected/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-kube-api-access-rqfww\") pod \"autores-model-46da77-kserve-5f56fd66c-m8z9v\" (UID: \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\") " pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:14:44.802731 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.802681    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-model-cache\") pod \"autores-model-46da77-kserve-5f56fd66c-m8z9v\" (UID: \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\") " pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:14:44.803031 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.803001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-dshm\") pod \"autores-model-46da77-kserve-5f56fd66c-m8z9v\" (UID: \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\") " pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:14:44.803031 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.803023    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-home\") pod \"autores-model-46da77-kserve-5f56fd66c-m8z9v\" (UID: \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\") " pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:14:44.803031 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.803037    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-model-cache\") pod \"autores-model-46da77-kserve-5f56fd66c-m8z9v\" (UID: \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\") " pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:14:44.803227 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.803005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-tmp-dir\") pod \"autores-model-46da77-kserve-5f56fd66c-m8z9v\" (UID: \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\") " pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:14:44.805420 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.805398    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-dshm\") pod \"autores-model-46da77-kserve-5f56fd66c-m8z9v\" (UID: \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\") " pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:14:44.805630 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.805614    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-tls-certs\") pod \"autores-model-46da77-kserve-5f56fd66c-m8z9v\" (UID: \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\") " pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:14:44.813914 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.813883    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqfww\" (UniqueName: \"kubernetes.io/projected/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-kube-api-access-rqfww\") pod \"autores-model-46da77-kserve-5f56fd66c-m8z9v\" (UID: \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\") " pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:14:44.901044 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.900953    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:14:44.903782 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.903755    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ee0409-c96d-4bb6-a55a-4dd23666154c-maas-api-service-ca\") pod \"09ee0409-c96d-4bb6-a55a-4dd23666154c\" (UID: \"09ee0409-c96d-4bb6-a55a-4dd23666154c\") "
Sep 11 20:14:44.904111 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:44.904087    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ee0409-c96d-4bb6-a55a-4dd23666154c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "09ee0409-c96d-4bb6-a55a-4dd23666154c" (UID: "09ee0409-c96d-4bb6-a55a-4dd23666154c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:14:45.005403 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:45.005369    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ee0409-c96d-4bb6-a55a-4dd23666154c-maas-api-service-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:14:45.181591 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:45.181511    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"]
Sep 11 20:14:45.185150 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:14:45.185120    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd3347ef_ee25_43d7_83e3_d9b3c1dfc51d.slice/crio-15c93af0cebaed1116e0aa924a89f4aaa969f31ec979a71d944e36e5af8a6ec3 WatchSource:0}: Error finding container 15c93af0cebaed1116e0aa924a89f4aaa969f31ec979a71d944e36e5af8a6ec3: Status 404 returned error can't find the container with id 15c93af0cebaed1116e0aa924a89f4aaa969f31ec979a71d944e36e5af8a6ec3
Sep 11 20:14:45.584484 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:45.584450    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cd58ce-pjfzn"
Sep 11 20:14:45.584484 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:45.584483    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cd58ce-pjfzn" event={"ID":"09ee0409-c96d-4bb6-a55a-4dd23666154c","Type":"ContainerDied","Data":"8fdf7f6346e4686020f618475269e2b5fe903603cc7cb154e26cb64e909e8b49"}
Sep 11 20:14:45.584757 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:45.584546    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fdf7f6346e4686020f618475269e2b5fe903603cc7cb154e26cb64e909e8b49"
Sep 11 20:14:45.586464 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:45.586436    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v" event={"ID":"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d","Type":"ContainerStarted","Data":"ae220ccb8aa246232150ce3b191ff34e7029fce0fdcaff40f550693df013ab90"}
Sep 11 20:14:45.586912 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:45.586473    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v" event={"ID":"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d","Type":"ContainerStarted","Data":"15c93af0cebaed1116e0aa924a89f4aaa969f31ec979a71d944e36e5af8a6ec3"}
Sep 11 20:14:45.607044 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:45.606996    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v" podStartSLOduration=1.6069802389999999 podStartE2EDuration="1.606980239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:14:45.604059721 +0000 UTC m=+855.596221991" watchObservedRunningTime="2026-09-11 20:14:45.606980239 +0000 UTC m=+855.599142508"
Sep 11 20:14:47.910546 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:47.910511    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cd58ce-pjfzn"]
Sep 11 20:14:47.912802 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:47.912780    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-16cd58ce-pjfzn"]
Sep 11 20:14:48.440956 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:48.440915    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09ee0409-c96d-4bb6-a55a-4dd23666154c" path="/var/lib/kubelet/pods/09ee0409-c96d-4bb6-a55a-4dd23666154c/volumes"
Sep 11 20:14:49.261810 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:49.261775    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:14:49.261810 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:49.261814    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:14:49.274400 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:49.274373    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:14:49.625183 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:49.625101    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:14:54.901464 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:54.901426    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:14:54.901871 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:54.901483    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:14:54.913485 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:54.913457    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:14:55.657677 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:55.657649    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:14:55.815094 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:14:55.815058    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:15:00.152241 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.152197    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e1dd-29819295-rj4cn"]
Sep 11 20:15:00.152826 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.152807    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09ee0409-c96d-4bb6-a55a-4dd23666154c" containerName="revoke-keys"
Sep 11 20:15:00.152871 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.152830    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="09ee0409-c96d-4bb6-a55a-4dd23666154c" containerName="revoke-keys"
Sep 11 20:15:00.152954 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.152943    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="09ee0409-c96d-4bb6-a55a-4dd23666154c" containerName="revoke-keys"
Sep 11 20:15:00.156309 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.156288    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f9558-29819295-vzdww"]
Sep 11 20:15:00.156411 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.156391    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e1dd-29819295-rj4cn"
Sep 11 20:15:00.158811 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.158790    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7dbs5\""
Sep 11 20:15:00.160166 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.160143    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9d9abb-29819295-8hrtp"]
Sep 11 20:15:00.160278 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.160246    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f9558-29819295-vzdww"
Sep 11 20:15:00.163948 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.163923    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9d9abb-29819295-8hrtp"
Sep 11 20:15:00.164273 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.164052    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e1dd-29819295-rj4cn"]
Sep 11 20:15:00.167326 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.167307    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f9558-29819295-vzdww"]
Sep 11 20:15:00.170341 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.170318    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9d9abb-29819295-8hrtp"]
Sep 11 20:15:00.171553 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.171532    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e1dd-29819295-rj4cn"
Sep 11 20:15:00.182376 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.182357    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46da77-29819295-d7lnv"]
Sep 11 20:15:00.182465 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.182413    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f9558-29819295-vzdww"
Sep 11 20:15:00.185784 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.185768    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46da77-29819295-d7lnv"
Sep 11 20:15:00.197130 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.196436    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46da77-29819295-d7lnv"]
Sep 11 20:15:00.224580 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.224547    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9d9abb-29819295-8hrtp"
Sep 11 20:15:00.232111 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.232087    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46da77-29819295-d7lnv"
Sep 11 20:15:00.465981 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.465955    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e1dd-29819295-rj4cn"]
Sep 11 20:15:00.468098 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:15:00.468067    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c995bc8_6b06_4897_a745_7e19aa68441b.slice/crio-c9d0d4e4dd8e32da59e3309438e3bd7135f5256447a37d6a18995fc33daf1f9b WatchSource:0}: Error finding container c9d0d4e4dd8e32da59e3309438e3bd7135f5256447a37d6a18995fc33daf1f9b: Status 404 returned error can't find the container with id c9d0d4e4dd8e32da59e3309438e3bd7135f5256447a37d6a18995fc33daf1f9b
Sep 11 20:15:00.482927 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.482903    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f9558-29819295-vzdww"]
Sep 11 20:15:00.484380 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:15:00.484351    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc0a542_ae11_4b7f_ad11_8431ac73871c.slice/crio-b59858ab8e73dd31bf2825445b9f8395765d527ce92b0698a063a7032b0f55f4 WatchSource:0}: Error finding container b59858ab8e73dd31bf2825445b9f8395765d527ce92b0698a063a7032b0f55f4: Status 404 returned error can't find the container with id b59858ab8e73dd31bf2825445b9f8395765d527ce92b0698a063a7032b0f55f4
Sep 11 20:15:00.532793 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.532766    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9d9abb-29819295-8hrtp"]
Sep 11 20:15:00.533105 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:15:00.533075    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b47b9a_3452_4feb_8841_f0180ea4e647.slice/crio-ebcebf935d260cd383e95cca3c071aa48749c315219e58f88a0c6665160d8e11 WatchSource:0}: Error finding container ebcebf935d260cd383e95cca3c071aa48749c315219e58f88a0c6665160d8e11: Status 404 returned error can't find the container with id ebcebf935d260cd383e95cca3c071aa48749c315219e58f88a0c6665160d8e11
Sep 11 20:15:00.535457 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.535434    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46da77-29819295-d7lnv"]
Sep 11 20:15:00.536274 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:15:00.536255    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c88529_8c7c_47c0_aa38_263494d0ee9e.slice/crio-ee9e03a00c90209a0305226f436c9e366785fecf91f899c18622f48ed190c05e WatchSource:0}: Error finding container ee9e03a00c90209a0305226f436c9e366785fecf91f899c18622f48ed190c05e: Status 404 returned error can't find the container with id ee9e03a00c90209a0305226f436c9e366785fecf91f899c18622f48ed190c05e
Sep 11 20:15:00.674242 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.674206    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9d9abb-29819295-8hrtp" event={"ID":"42b47b9a-3452-4feb-8841-f0180ea4e647","Type":"ContainerStarted","Data":"ebcebf935d260cd383e95cca3c071aa48749c315219e58f88a0c6665160d8e11"}
Sep 11 20:15:00.675466 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.675437    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f9558-29819295-vzdww" event={"ID":"ecc0a542-ae11-4b7f-ad11-8431ac73871c","Type":"ContainerStarted","Data":"b59858ab8e73dd31bf2825445b9f8395765d527ce92b0698a063a7032b0f55f4"}
Sep 11 20:15:00.676650 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.676625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e1dd-29819295-rj4cn" event={"ID":"2c995bc8-6b06-4897-a745-7e19aa68441b","Type":"ContainerStarted","Data":"c9d0d4e4dd8e32da59e3309438e3bd7135f5256447a37d6a18995fc33daf1f9b"}
Sep 11 20:15:00.677701 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:00.677680    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46da77-29819295-d7lnv" event={"ID":"40c88529-8c7c-47c0-aa38-263494d0ee9e","Type":"ContainerStarted","Data":"ee9e03a00c90209a0305226f436c9e366785fecf91f899c18622f48ed190c05e"}
Sep 11 20:15:01.684604 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:01.684543    2580 generic.go:356] "Generic (PLEG): container finished" podID="ecc0a542-ae11-4b7f-ad11-8431ac73871c" containerID="cfc3845478758cd4a7adf263d302cc8f2a9d65a72a7a1cb4e276ddf62f5b34ee" exitCode=0
Sep 11 20:15:01.684967 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:01.684638    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f9558-29819295-vzdww" event={"ID":"ecc0a542-ae11-4b7f-ad11-8431ac73871c","Type":"ContainerDied","Data":"cfc3845478758cd4a7adf263d302cc8f2a9d65a72a7a1cb4e276ddf62f5b34ee"}
Sep 11 20:15:01.686330 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:01.686310    2580 generic.go:356] "Generic (PLEG): container finished" podID="2c995bc8-6b06-4897-a745-7e19aa68441b" containerID="19926e0307a82c7f5109de36b9c294d8e16c7c8ebd6ac9a6c529ac3ef808e705" exitCode=0
Sep 11 20:15:01.686424 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:01.686385    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e1dd-29819295-rj4cn" event={"ID":"2c995bc8-6b06-4897-a745-7e19aa68441b","Type":"ContainerDied","Data":"19926e0307a82c7f5109de36b9c294d8e16c7c8ebd6ac9a6c529ac3ef808e705"}
Sep 11 20:15:01.687894 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:01.687869    2580 generic.go:356] "Generic (PLEG): container finished" podID="40c88529-8c7c-47c0-aa38-263494d0ee9e" containerID="61155eba929bf03cca696ab03ad560bc3db6a46ff39373d9ce85e06622a008e2" exitCode=0
Sep 11 20:15:01.687989 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:01.687944    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46da77-29819295-d7lnv" event={"ID":"40c88529-8c7c-47c0-aa38-263494d0ee9e","Type":"ContainerDied","Data":"61155eba929bf03cca696ab03ad560bc3db6a46ff39373d9ce85e06622a008e2"}
Sep 11 20:15:01.689344 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:01.689325    2580 generic.go:356] "Generic (PLEG): container finished" podID="42b47b9a-3452-4feb-8841-f0180ea4e647" containerID="1cca8c26c090ed614bb621b9bf993fedcc6a0aa4f1289782a67fae621f33766f" exitCode=0
Sep 11 20:15:01.689433 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:01.689354    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9d9abb-29819295-8hrtp" event={"ID":"42b47b9a-3452-4feb-8841-f0180ea4e647","Type":"ContainerDied","Data":"1cca8c26c090ed614bb621b9bf993fedcc6a0aa4f1289782a67fae621f33766f"}
Sep 11 20:15:02.834725 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:02.834700    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f9558-29819295-vzdww"
Sep 11 20:15:02.887472 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:02.887445    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9d9abb-29819295-8hrtp"
Sep 11 20:15:02.889935 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:02.889917    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e1dd-29819295-rj4cn"
Sep 11 20:15:02.910938 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:02.910908    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46da77-29819295-d7lnv"
Sep 11 20:15:03.702847 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:03.702815    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e1dd-29819295-rj4cn"
Sep 11 20:15:03.703043 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:03.702847    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e1dd-29819295-rj4cn" event={"ID":"2c995bc8-6b06-4897-a745-7e19aa68441b","Type":"ContainerDied","Data":"c9d0d4e4dd8e32da59e3309438e3bd7135f5256447a37d6a18995fc33daf1f9b"}
Sep 11 20:15:03.703043 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:03.702894    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9d0d4e4dd8e32da59e3309438e3bd7135f5256447a37d6a18995fc33daf1f9b"
Sep 11 20:15:03.704152 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:03.704134    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46da77-29819295-d7lnv"
Sep 11 20:15:03.704285 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:03.704164    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46da77-29819295-d7lnv" event={"ID":"40c88529-8c7c-47c0-aa38-263494d0ee9e","Type":"ContainerDied","Data":"ee9e03a00c90209a0305226f436c9e366785fecf91f899c18622f48ed190c05e"}
Sep 11 20:15:03.704285 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:03.704213    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee9e03a00c90209a0305226f436c9e366785fecf91f899c18622f48ed190c05e"
Sep 11 20:15:03.705614 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:03.705484    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9d9abb-29819295-8hrtp"
Sep 11 20:15:03.705614 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:03.705526    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9d9abb-29819295-8hrtp" event={"ID":"42b47b9a-3452-4feb-8841-f0180ea4e647","Type":"ContainerDied","Data":"ebcebf935d260cd383e95cca3c071aa48749c315219e58f88a0c6665160d8e11"}
Sep 11 20:15:03.705614 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:03.705584    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebcebf935d260cd383e95cca3c071aa48749c315219e58f88a0c6665160d8e11"
Sep 11 20:15:03.706956 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:03.706934    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f9558-29819295-vzdww" event={"ID":"ecc0a542-ae11-4b7f-ad11-8431ac73871c","Type":"ContainerDied","Data":"b59858ab8e73dd31bf2825445b9f8395765d527ce92b0698a063a7032b0f55f4"}
Sep 11 20:15:03.707048 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:03.706955    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f9558-29819295-vzdww"
Sep 11 20:15:03.707048 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:03.706976    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b59858ab8e73dd31bf2825445b9f8395765d527ce92b0698a063a7032b0f55f4"
Sep 11 20:15:05.673726 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.673692    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"]
Sep 11 20:15:05.674267 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.674246    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b47b9a-3452-4feb-8841-f0180ea4e647" containerName="cleanup"
Sep 11 20:15:05.674369 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.674270    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b47b9a-3452-4feb-8841-f0180ea4e647" containerName="cleanup"
Sep 11 20:15:05.674369 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.674297    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c995bc8-6b06-4897-a745-7e19aa68441b" containerName="cleanup"
Sep 11 20:15:05.674369 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.674307    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c995bc8-6b06-4897-a745-7e19aa68441b" containerName="cleanup"
Sep 11 20:15:05.674369 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.674321    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecc0a542-ae11-4b7f-ad11-8431ac73871c" containerName="cleanup"
Sep 11 20:15:05.674369 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.674329    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecc0a542-ae11-4b7f-ad11-8431ac73871c" containerName="cleanup"
Sep 11 20:15:05.674369 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.674339    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40c88529-8c7c-47c0-aa38-263494d0ee9e" containerName="cleanup"
Sep 11 20:15:05.674369 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.674347    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="40c88529-8c7c-47c0-aa38-263494d0ee9e" containerName="cleanup"
Sep 11 20:15:05.674718 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.674443    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c995bc8-6b06-4897-a745-7e19aa68441b" containerName="cleanup"
Sep 11 20:15:05.674718 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.674456    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="42b47b9a-3452-4feb-8841-f0180ea4e647" containerName="cleanup"
Sep 11 20:15:05.674718 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.674467    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="40c88529-8c7c-47c0-aa38-263494d0ee9e" containerName="cleanup"
Sep 11 20:15:05.674718 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.674478    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecc0a542-ae11-4b7f-ad11-8431ac73871c" containerName="cleanup"
Sep 11 20:15:05.677669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.677649    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:15:05.680377 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.680359    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b3e1dd\"/\"openshift-service-ca.crt\""
Sep 11 20:15:05.681304 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.681281    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b3e1dd\"/\"default-dockercfg-thdj7\""
Sep 11 20:15:05.681304 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.681294    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b3e1dd\"/\"test-model-b3e1dd-kserve-self-signed-certs\""
Sep 11 20:15:05.681454 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.681321    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b3e1dd\"/\"kube-root-ca.crt\""
Sep 11 20:15:05.687611 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.687561    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"]
Sep 11 20:15:05.689764 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.689415    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/048e46f1-7821-4ecd-8cb4-93b091672c59-tmp-dir\") pod \"test-model-b3e1dd-kserve-79d5488dcf-lk6s9\" (UID: \"048e46f1-7821-4ecd-8cb4-93b091672c59\") " pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:15:05.689764 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.689455    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/048e46f1-7821-4ecd-8cb4-93b091672c59-tls-certs\") pod \"test-model-b3e1dd-kserve-79d5488dcf-lk6s9\" (UID: \"048e46f1-7821-4ecd-8cb4-93b091672c59\") " pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:15:05.689764 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.689536    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/048e46f1-7821-4ecd-8cb4-93b091672c59-dshm\") pod \"test-model-b3e1dd-kserve-79d5488dcf-lk6s9\" (UID: \"048e46f1-7821-4ecd-8cb4-93b091672c59\") " pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:15:05.689764 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.689575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/048e46f1-7821-4ecd-8cb4-93b091672c59-home\") pod \"test-model-b3e1dd-kserve-79d5488dcf-lk6s9\" (UID: \"048e46f1-7821-4ecd-8cb4-93b091672c59\") " pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:15:05.689764 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.689603    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/048e46f1-7821-4ecd-8cb4-93b091672c59-model-cache\") pod \"test-model-b3e1dd-kserve-79d5488dcf-lk6s9\" (UID: \"048e46f1-7821-4ecd-8cb4-93b091672c59\") " pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:15:05.689764 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.689631    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnfjw\" (UniqueName: \"kubernetes.io/projected/048e46f1-7821-4ecd-8cb4-93b091672c59-kube-api-access-gnfjw\") pod \"test-model-b3e1dd-kserve-79d5488dcf-lk6s9\" (UID: \"048e46f1-7821-4ecd-8cb4-93b091672c59\") " pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:15:05.791173 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.791128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/048e46f1-7821-4ecd-8cb4-93b091672c59-model-cache\") pod \"test-model-b3e1dd-kserve-79d5488dcf-lk6s9\" (UID: \"048e46f1-7821-4ecd-8cb4-93b091672c59\") " pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:15:05.791455 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.791423    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnfjw\" (UniqueName: \"kubernetes.io/projected/048e46f1-7821-4ecd-8cb4-93b091672c59-kube-api-access-gnfjw\") pod \"test-model-b3e1dd-kserve-79d5488dcf-lk6s9\" (UID: \"048e46f1-7821-4ecd-8cb4-93b091672c59\") " pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:15:05.791678 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.791658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/048e46f1-7821-4ecd-8cb4-93b091672c59-tmp-dir\") pod \"test-model-b3e1dd-kserve-79d5488dcf-lk6s9\" (UID: \"048e46f1-7821-4ecd-8cb4-93b091672c59\") " pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:15:05.791792 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.791778    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/048e46f1-7821-4ecd-8cb4-93b091672c59-tls-certs\") pod \"test-model-b3e1dd-kserve-79d5488dcf-lk6s9\" (UID: \"048e46f1-7821-4ecd-8cb4-93b091672c59\") " pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:15:05.791943 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.791930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/048e46f1-7821-4ecd-8cb4-93b091672c59-dshm\") pod \"test-model-b3e1dd-kserve-79d5488dcf-lk6s9\" (UID: \"048e46f1-7821-4ecd-8cb4-93b091672c59\") " pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:15:05.792077 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.792063    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/048e46f1-7821-4ecd-8cb4-93b091672c59-home\") pod \"test-model-b3e1dd-kserve-79d5488dcf-lk6s9\" (UID: \"048e46f1-7821-4ecd-8cb4-93b091672c59\") " pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:15:05.792260 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.792235    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/048e46f1-7821-4ecd-8cb4-93b091672c59-tmp-dir\") pod \"test-model-b3e1dd-kserve-79d5488dcf-lk6s9\" (UID: \"048e46f1-7821-4ecd-8cb4-93b091672c59\") " pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:15:05.792628 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.792609    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/048e46f1-7821-4ecd-8cb4-93b091672c59-home\") pod \"test-model-b3e1dd-kserve-79d5488dcf-lk6s9\" (UID: \"048e46f1-7821-4ecd-8cb4-93b091672c59\") " pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:15:05.792762 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.791998    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/048e46f1-7821-4ecd-8cb4-93b091672c59-model-cache\") pod \"test-model-b3e1dd-kserve-79d5488dcf-lk6s9\" (UID: \"048e46f1-7821-4ecd-8cb4-93b091672c59\") " pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:15:05.795816 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.795791    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/048e46f1-7821-4ecd-8cb4-93b091672c59-tls-certs\") pod \"test-model-b3e1dd-kserve-79d5488dcf-lk6s9\" (UID: \"048e46f1-7821-4ecd-8cb4-93b091672c59\") " pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:15:05.801273 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.801251    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/048e46f1-7821-4ecd-8cb4-93b091672c59-dshm\") pod \"test-model-b3e1dd-kserve-79d5488dcf-lk6s9\" (UID: \"048e46f1-7821-4ecd-8cb4-93b091672c59\") " pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:15:05.801403 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.801384    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnfjw\" (UniqueName: \"kubernetes.io/projected/048e46f1-7821-4ecd-8cb4-93b091672c59-kube-api-access-gnfjw\") pod \"test-model-b3e1dd-kserve-79d5488dcf-lk6s9\" (UID: \"048e46f1-7821-4ecd-8cb4-93b091672c59\") " pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:15:05.989093 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:05.989048    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:15:06.291166 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:06.291003    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"]
Sep 11 20:15:06.294036 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:15:06.294007    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod048e46f1_7821_4ecd_8cb4_93b091672c59.slice/crio-c985846dac843cf1a9a5f6b0de813c0b9c4e4b13fbcbe3bcafa936af2c83c4ea WatchSource:0}: Error finding container c985846dac843cf1a9a5f6b0de813c0b9c4e4b13fbcbe3bcafa936af2c83c4ea: Status 404 returned error can't find the container with id c985846dac843cf1a9a5f6b0de813c0b9c4e4b13fbcbe3bcafa936af2c83c4ea
Sep 11 20:15:06.729577 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:06.729542    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9" event={"ID":"048e46f1-7821-4ecd-8cb4-93b091672c59","Type":"ContainerStarted","Data":"7dbbde7a5515cb64db2e3406be19c023837c3bdc1de3ac0bec1e2df37f698a06"}
Sep 11 20:15:06.729577 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:06.729583    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9" event={"ID":"048e46f1-7821-4ecd-8cb4-93b091672c59","Type":"ContainerStarted","Data":"c985846dac843cf1a9a5f6b0de813c0b9c4e4b13fbcbe3bcafa936af2c83c4ea"}
Sep 11 20:15:06.749721 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:06.749666    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9" podStartSLOduration=1.749650183 podStartE2EDuration="1.749650183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:15:06.746866076 +0000 UTC m=+876.739028347" watchObservedRunningTime="2026-09-11 20:15:06.749650183 +0000 UTC m=+876.741812478"
Sep 11 20:15:15.912548 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:15.912510    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:15:15.990240 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:15.990210    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:15:15.990413 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:15.990252    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:15:16.015369 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:16.015342    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:15:16.810016 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:16.809989    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:15:16.870633 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:16.870598    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5b4f720-x5qlf"]
Sep 11 20:15:16.875151 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:16.875129    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5b4f720-x5qlf"
Sep 11 20:15:16.877854 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:16.877830    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7dbs5\""
Sep 11 20:15:16.877977 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:16.877942    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:15:16.881875 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:16.881853    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5b4f720-x5qlf"]
Sep 11 20:15:16.994272 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:16.994239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85b5ff62-5633-4de3-a7aa-669850db5404-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f5b4f720-x5qlf\" (UID: \"85b5ff62-5633-4de3-a7aa-669850db5404\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5b4f720-x5qlf"
Sep 11 20:15:17.095635 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:17.095550    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85b5ff62-5633-4de3-a7aa-669850db5404-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f5b4f720-x5qlf\" (UID: \"85b5ff62-5633-4de3-a7aa-669850db5404\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5b4f720-x5qlf"
Sep 11 20:15:17.096181 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:17.096153    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85b5ff62-5633-4de3-a7aa-669850db5404-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f5b4f720-x5qlf\" (UID: \"85b5ff62-5633-4de3-a7aa-669850db5404\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5b4f720-x5qlf"
Sep 11 20:15:17.206221 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:17.206190    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5b4f720-x5qlf"
Sep 11 20:15:17.477618 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:17.477587    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5b4f720-x5qlf"]
Sep 11 20:15:17.479071 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:15:17.479046    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b5ff62_5633_4de3_a7aa_669850db5404.slice/crio-6f43706780edbb5de633772701eeb8f05d40f4f7333b4240882dea4fb0b358cf WatchSource:0}: Error finding container 6f43706780edbb5de633772701eeb8f05d40f4f7333b4240882dea4fb0b358cf: Status 404 returned error can't find the container with id 6f43706780edbb5de633772701eeb8f05d40f4f7333b4240882dea4fb0b358cf
Sep 11 20:15:17.802299 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:17.802265    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5b4f720-x5qlf" event={"ID":"85b5ff62-5633-4de3-a7aa-669850db5404","Type":"ContainerStarted","Data":"f131875f48e0fb3924a32b460905a90c501acb49cd8773a99c1d9af93a57703d"}
Sep 11 20:15:17.802451 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:17.802316    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5b4f720-x5qlf" event={"ID":"85b5ff62-5633-4de3-a7aa-669850db5404","Type":"ContainerStarted","Data":"6f43706780edbb5de633772701eeb8f05d40f4f7333b4240882dea4fb0b358cf"}
Sep 11 20:15:17.819899 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:17.819858    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5b4f720-x5qlf" podStartSLOduration=1.819843053 podStartE2EDuration="1.819843053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:15:17.817534769 +0000 UTC m=+887.809697039" watchObservedRunningTime="2026-09-11 20:15:17.819843053 +0000 UTC m=+887.812005323"
Sep 11 20:15:18.809985 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:18.809951    2580 generic.go:356] "Generic (PLEG): container finished" podID="85b5ff62-5633-4de3-a7aa-669850db5404" containerID="f131875f48e0fb3924a32b460905a90c501acb49cd8773a99c1d9af93a57703d" exitCode=0
Sep 11 20:15:18.810411 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:18.810030    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5b4f720-x5qlf" event={"ID":"85b5ff62-5633-4de3-a7aa-669850db5404","Type":"ContainerDied","Data":"f131875f48e0fb3924a32b460905a90c501acb49cd8773a99c1d9af93a57703d"}
Sep 11 20:15:19.901049 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:19.901020    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46da77-4x6cm"]
Sep 11 20:15:19.904731 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:19.904712    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46da77-4x6cm"
Sep 11 20:15:19.913650 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:19.913365    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46da77-4x6cm"]
Sep 11 20:15:19.951108 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:19.951088    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5b4f720-x5qlf"
Sep 11 20:15:20.019995 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:20.019957    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85b5ff62-5633-4de3-a7aa-669850db5404-maas-api-service-ca\") pod \"85b5ff62-5633-4de3-a7aa-669850db5404\" (UID: \"85b5ff62-5633-4de3-a7aa-669850db5404\") "
Sep 11 20:15:20.020171 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:20.020072    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e21dd750-305f-48de-84ae-50221476125d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-46da77-4x6cm\" (UID: \"e21dd750-305f-48de-84ae-50221476125d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46da77-4x6cm"
Sep 11 20:15:20.020358 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:20.020335    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b5ff62-5633-4de3-a7aa-669850db5404-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "85b5ff62-5633-4de3-a7aa-669850db5404" (UID: "85b5ff62-5633-4de3-a7aa-669850db5404"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:15:20.121630 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:20.121536    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e21dd750-305f-48de-84ae-50221476125d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-46da77-4x6cm\" (UID: \"e21dd750-305f-48de-84ae-50221476125d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46da77-4x6cm"
Sep 11 20:15:20.121777 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:20.121668    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85b5ff62-5633-4de3-a7aa-669850db5404-maas-api-service-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:15:20.122150 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:20.122129    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e21dd750-305f-48de-84ae-50221476125d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-46da77-4x6cm\" (UID: \"e21dd750-305f-48de-84ae-50221476125d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46da77-4x6cm"
Sep 11 20:15:20.218581 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:20.218555    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46da77-4x6cm"
Sep 11 20:15:20.489906 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:20.489877    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46da77-4x6cm"]
Sep 11 20:15:20.490932 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:15:20.490903    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21dd750_305f_48de_84ae_50221476125d.slice/crio-bca6458e7fc539477117324351f3cc09e0e0d767189eb07a0deee528be36bc12 WatchSource:0}: Error finding container bca6458e7fc539477117324351f3cc09e0e0d767189eb07a0deee528be36bc12: Status 404 returned error can't find the container with id bca6458e7fc539477117324351f3cc09e0e0d767189eb07a0deee528be36bc12
Sep 11 20:15:20.822802 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:20.822761    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46da77-4x6cm" event={"ID":"e21dd750-305f-48de-84ae-50221476125d","Type":"ContainerStarted","Data":"134e0d3718724dac1021549774c3828c6c213e1b18db59afd0abd883b9ac974b"}
Sep 11 20:15:20.822954 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:20.822866    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46da77-4x6cm" event={"ID":"e21dd750-305f-48de-84ae-50221476125d","Type":"ContainerStarted","Data":"bca6458e7fc539477117324351f3cc09e0e0d767189eb07a0deee528be36bc12"}
Sep 11 20:15:20.824038 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:20.824020    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5b4f720-x5qlf"
Sep 11 20:15:20.824130 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:20.824052    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5b4f720-x5qlf" event={"ID":"85b5ff62-5633-4de3-a7aa-669850db5404","Type":"ContainerDied","Data":"6f43706780edbb5de633772701eeb8f05d40f4f7333b4240882dea4fb0b358cf"}
Sep 11 20:15:20.824130 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:20.824096    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f43706780edbb5de633772701eeb8f05d40f4f7333b4240882dea4fb0b358cf"
Sep 11 20:15:20.840630 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:20.840589    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46da77-4x6cm" podStartSLOduration=1.840575588 podStartE2EDuration="1.840575588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:15:20.838858714 +0000 UTC m=+890.831020987" watchObservedRunningTime="2026-09-11 20:15:20.840575588 +0000 UTC m=+890.832737857"
Sep 11 20:15:21.831117 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:21.831084    2580 generic.go:356] "Generic (PLEG): container finished" podID="e21dd750-305f-48de-84ae-50221476125d" containerID="134e0d3718724dac1021549774c3828c6c213e1b18db59afd0abd883b9ac974b" exitCode=0
Sep 11 20:15:21.831458 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:21.831144    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46da77-4x6cm" event={"ID":"e21dd750-305f-48de-84ae-50221476125d","Type":"ContainerDied","Data":"134e0d3718724dac1021549774c3828c6c213e1b18db59afd0abd883b9ac974b"}
Sep 11 20:15:22.968718 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:22.968693    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46da77-4x6cm"
Sep 11 20:15:23.045138 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:23.045108    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e21dd750-305f-48de-84ae-50221476125d-maas-api-service-ca\") pod \"e21dd750-305f-48de-84ae-50221476125d\" (UID: \"e21dd750-305f-48de-84ae-50221476125d\") "
Sep 11 20:15:23.045480 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:23.045455    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e21dd750-305f-48de-84ae-50221476125d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e21dd750-305f-48de-84ae-50221476125d" (UID: "e21dd750-305f-48de-84ae-50221476125d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:15:23.145879 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:23.145794    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e21dd750-305f-48de-84ae-50221476125d-maas-api-service-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:15:23.844678 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:23.844652    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46da77-4x6cm"
Sep 11 20:15:23.844851 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:23.844656    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46da77-4x6cm" event={"ID":"e21dd750-305f-48de-84ae-50221476125d","Type":"ContainerDied","Data":"bca6458e7fc539477117324351f3cc09e0e0d767189eb07a0deee528be36bc12"}
Sep 11 20:15:23.844851 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:23.844782    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bca6458e7fc539477117324351f3cc09e0e0d767189eb07a0deee528be36bc12"
Sep 11 20:15:26.531911 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:26.531879    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5b4f720-x5qlf"]
Sep 11 20:15:26.534207 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:26.534186    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f5b4f720-x5qlf"]
Sep 11 20:15:28.441159 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:28.441126    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85b5ff62-5633-4de3-a7aa-669850db5404" path="/var/lib/kubelet/pods/85b5ff62-5633-4de3-a7aa-669850db5404/volumes"
Sep 11 20:15:30.145670 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.145623    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46da77-29819295-d7lnv"]
Sep 11 20:15:30.147954 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.147933    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46da77-29819295-d7lnv"]
Sep 11 20:15:30.168873 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.168843    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr"]
Sep 11 20:15:30.169100 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.169073    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr" podUID="abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf" containerName="payload-processing" containerID="cri-o://e784143745d2a6574083b0f16207807a96759b8373f4f3950eca31d03da5e96a" gracePeriod=30
Sep 11 20:15:30.178782 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.178754    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9"]
Sep 11 20:15:30.179002 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.178977    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9" podUID="eeb00a15-0884-4dee-9ff4-5a444d79a148" containerName="payload-pre-processing" containerID="cri-o://3054b8b256905d00ebed159fd0f24e603a3efcf4e358d204d82f4aed3c0967dc" gracePeriod=30
Sep 11 20:15:30.439353 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.439319    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr"
Sep 11 20:15:30.443670 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.443641    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40c88529-8c7c-47c0-aa38-263494d0ee9e" path="/var/lib/kubelet/pods/40c88529-8c7c-47c0-aa38-263494d0ee9e/volumes"
Sep 11 20:15:30.470653 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.470632    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9"
Sep 11 20:15:30.511085 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.511057    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf-plugins-config-volume\") pod \"abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf\" (UID: \"abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf\") "
Sep 11 20:15:30.511227 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.511098    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fs5xf\" (UniqueName: \"kubernetes.io/projected/abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf-kube-api-access-fs5xf\") pod \"abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf\" (UID: \"abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf\") "
Sep 11 20:15:30.511467 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.511441    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf" (UID: "abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:15:30.513139 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.513117    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf-kube-api-access-fs5xf" (OuterVolumeSpecName: "kube-api-access-fs5xf") pod "abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf" (UID: "abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf"). InnerVolumeSpecName "kube-api-access-fs5xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:15:30.551727 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.551694    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46da77-4x6cm"]
Sep 11 20:15:30.555948 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.555924    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46da77-4x6cm"]
Sep 11 20:15:30.611798 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.611768    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kf4cc\" (UniqueName: \"kubernetes.io/projected/eeb00a15-0884-4dee-9ff4-5a444d79a148-kube-api-access-kf4cc\") pod \"eeb00a15-0884-4dee-9ff4-5a444d79a148\" (UID: \"eeb00a15-0884-4dee-9ff4-5a444d79a148\") "
Sep 11 20:15:30.611952 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.611813    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eeb00a15-0884-4dee-9ff4-5a444d79a148-plugins-config-volume\") pod \"eeb00a15-0884-4dee-9ff4-5a444d79a148\" (UID: \"eeb00a15-0884-4dee-9ff4-5a444d79a148\") "
Sep 11 20:15:30.612047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.612028    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf-plugins-config-volume\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:15:30.612106 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.612054    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs5xf\" (UniqueName: \"kubernetes.io/projected/abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf-kube-api-access-fs5xf\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:15:30.612259 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.612236    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb00a15-0884-4dee-9ff4-5a444d79a148-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eeb00a15-0884-4dee-9ff4-5a444d79a148" (UID: "eeb00a15-0884-4dee-9ff4-5a444d79a148"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:15:30.614210 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.614186    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb00a15-0884-4dee-9ff4-5a444d79a148-kube-api-access-kf4cc" (OuterVolumeSpecName: "kube-api-access-kf4cc") pod "eeb00a15-0884-4dee-9ff4-5a444d79a148" (UID: "eeb00a15-0884-4dee-9ff4-5a444d79a148"). InnerVolumeSpecName "kube-api-access-kf4cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:15:30.713295 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.713258    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf4cc\" (UniqueName: \"kubernetes.io/projected/eeb00a15-0884-4dee-9ff4-5a444d79a148-kube-api-access-kf4cc\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:15:30.713295 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.713289    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eeb00a15-0884-4dee-9ff4-5a444d79a148-plugins-config-volume\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:15:30.885201 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.885166    2580 generic.go:356] "Generic (PLEG): container finished" podID="abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf" containerID="e784143745d2a6574083b0f16207807a96759b8373f4f3950eca31d03da5e96a" exitCode=0
Sep 11 20:15:30.885366 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.885213    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr"
Sep 11 20:15:30.885366 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.885259    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr" event={"ID":"abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf","Type":"ContainerDied","Data":"e784143745d2a6574083b0f16207807a96759b8373f4f3950eca31d03da5e96a"}
Sep 11 20:15:30.885366 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.885320    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr" event={"ID":"abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf","Type":"ContainerDied","Data":"120f799f00a6a0a5b65eb960453f37e595cdbc56d8400953a69346d2022c88f2"}
Sep 11 20:15:30.885366 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.885348    2580 scope.go:117] "RemoveContainer" containerID="e784143745d2a6574083b0f16207807a96759b8373f4f3950eca31d03da5e96a"
Sep 11 20:15:30.886659 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.886636    2580 generic.go:356] "Generic (PLEG): container finished" podID="eeb00a15-0884-4dee-9ff4-5a444d79a148" containerID="3054b8b256905d00ebed159fd0f24e603a3efcf4e358d204d82f4aed3c0967dc" exitCode=0
Sep 11 20:15:30.886778 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.886667    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9" event={"ID":"eeb00a15-0884-4dee-9ff4-5a444d79a148","Type":"ContainerDied","Data":"3054b8b256905d00ebed159fd0f24e603a3efcf4e358d204d82f4aed3c0967dc"}
Sep 11 20:15:30.886778 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.886684    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9"
Sep 11 20:15:30.886778 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.886718    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9" event={"ID":"eeb00a15-0884-4dee-9ff4-5a444d79a148","Type":"ContainerDied","Data":"99ab31a9d13b3b9a9dc0a0104675c121511e3f2d82963bdb9e6f606afec6f949"}
Sep 11 20:15:30.896286 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.896271    2580 scope.go:117] "RemoveContainer" containerID="e784143745d2a6574083b0f16207807a96759b8373f4f3950eca31d03da5e96a"
Sep 11 20:15:30.896653 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:15:30.896627    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e784143745d2a6574083b0f16207807a96759b8373f4f3950eca31d03da5e96a\": container with ID starting with e784143745d2a6574083b0f16207807a96759b8373f4f3950eca31d03da5e96a not found: ID does not exist" containerID="e784143745d2a6574083b0f16207807a96759b8373f4f3950eca31d03da5e96a"
Sep 11 20:15:30.896747 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.896665    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e784143745d2a6574083b0f16207807a96759b8373f4f3950eca31d03da5e96a"} err="failed to get container status \"e784143745d2a6574083b0f16207807a96759b8373f4f3950eca31d03da5e96a\": rpc error: code = NotFound desc = could not find container \"e784143745d2a6574083b0f16207807a96759b8373f4f3950eca31d03da5e96a\": container with ID starting with e784143745d2a6574083b0f16207807a96759b8373f4f3950eca31d03da5e96a not found: ID does not exist"
Sep 11 20:15:30.896747 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.896688    2580 scope.go:117] "RemoveContainer" containerID="3054b8b256905d00ebed159fd0f24e603a3efcf4e358d204d82f4aed3c0967dc"
Sep 11 20:15:30.905066 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.905048    2580 scope.go:117] "RemoveContainer" containerID="3054b8b256905d00ebed159fd0f24e603a3efcf4e358d204d82f4aed3c0967dc"
Sep 11 20:15:30.905356 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:15:30.905316    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3054b8b256905d00ebed159fd0f24e603a3efcf4e358d204d82f4aed3c0967dc\": container with ID starting with 3054b8b256905d00ebed159fd0f24e603a3efcf4e358d204d82f4aed3c0967dc not found: ID does not exist" containerID="3054b8b256905d00ebed159fd0f24e603a3efcf4e358d204d82f4aed3c0967dc"
Sep 11 20:15:30.905459 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.905364    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3054b8b256905d00ebed159fd0f24e603a3efcf4e358d204d82f4aed3c0967dc"} err="failed to get container status \"3054b8b256905d00ebed159fd0f24e603a3efcf4e358d204d82f4aed3c0967dc\": rpc error: code = NotFound desc = could not find container \"3054b8b256905d00ebed159fd0f24e603a3efcf4e358d204d82f4aed3c0967dc\": container with ID starting with 3054b8b256905d00ebed159fd0f24e603a3efcf4e358d204d82f4aed3c0967dc not found: ID does not exist"
Sep 11 20:15:30.908691 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.908670    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr"]
Sep 11 20:15:30.912926 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.912905    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-46da77-7df855d5f9-pcpfr"]
Sep 11 20:15:30.923475 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.923453    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9"]
Sep 11 20:15:30.929267 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:30.929246    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-46da77-c99645dc8-lnrq9"]
Sep 11 20:15:32.440543 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:32.440513    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf" path="/var/lib/kubelet/pods/abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf/volumes"
Sep 11 20:15:32.441022 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:32.440960    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e21dd750-305f-48de-84ae-50221476125d" path="/var/lib/kubelet/pods/e21dd750-305f-48de-84ae-50221476125d/volumes"
Sep 11 20:15:32.441333 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:32.441314    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeb00a15-0884-4dee-9ff4-5a444d79a148" path="/var/lib/kubelet/pods/eeb00a15-0884-4dee-9ff4-5a444d79a148/volumes"
Sep 11 20:15:35.826775 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:35.826742    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"]
Sep 11 20:15:36.108278 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:36.108192    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:15:50.845899 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:50.845839    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v" podUID="dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d" containerName="main" containerID="cri-o://ae220ccb8aa246232150ce3b191ff34e7029fce0fdcaff40f550693df013ab90" gracePeriod=45
Sep 11 20:15:51.006085 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:51.006050    2580 generic.go:356] "Generic (PLEG): container finished" podID="dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d" containerID="ae220ccb8aa246232150ce3b191ff34e7029fce0fdcaff40f550693df013ab90" exitCode=0
Sep 11 20:15:51.006257 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:51.006118    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v" event={"ID":"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d","Type":"ContainerDied","Data":"ae220ccb8aa246232150ce3b191ff34e7029fce0fdcaff40f550693df013ab90"}
Sep 11 20:15:51.100664 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:51.100608    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:15:51.197132 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:51.197102    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-tls-certs\") pod \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\" (UID: \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\") "
Sep 11 20:15:51.197132 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:51.197134    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqfww\" (UniqueName: \"kubernetes.io/projected/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-kube-api-access-rqfww\") pod \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\" (UID: \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\") "
Sep 11 20:15:51.197350 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:51.197169    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-home\") pod \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\" (UID: \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\") "
Sep 11 20:15:51.197350 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:51.197195    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-tmp-dir\") pod \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\" (UID: \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\") "
Sep 11 20:15:51.197350 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:51.197231    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-model-cache\") pod \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\" (UID: \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\") "
Sep 11 20:15:51.197350 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:51.197245    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-dshm\") pod \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\" (UID: \"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d\") "
Sep 11 20:15:51.197582 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:51.197521    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d" (UID: "dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:15:51.197582 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:51.197556    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-home" (OuterVolumeSpecName: "home") pod "dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d" (UID: "dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:15:51.197582 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:51.197567    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-model-cache" (OuterVolumeSpecName: "model-cache") pod "dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d" (UID: "dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:15:51.199393 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:51.199368    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-kube-api-access-rqfww" (OuterVolumeSpecName: "kube-api-access-rqfww") pod "dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d" (UID: "dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d"). InnerVolumeSpecName "kube-api-access-rqfww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:15:51.199515 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:51.199395    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-dshm" (OuterVolumeSpecName: "dshm") pod "dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d" (UID: "dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:15:51.199515 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:51.199426    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d" (UID: "dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:15:51.298265 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:51.298238    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-model-cache\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:15:51.298265 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:51.298261    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-dshm\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:15:51.298265 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:51.298270    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-tls-certs\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:15:51.298451 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:51.298278    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqfww\" (UniqueName: \"kubernetes.io/projected/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-kube-api-access-rqfww\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:15:51.298451 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:51.298288    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-home\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:15:51.298451 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:51.298316    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d-tmp-dir\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:15:52.013522 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:52.013475    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"
Sep 11 20:15:52.013993 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:52.013475    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v" event={"ID":"dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d","Type":"ContainerDied","Data":"15c93af0cebaed1116e0aa924a89f4aaa969f31ec979a71d944e36e5af8a6ec3"}
Sep 11 20:15:52.013993 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:52.013609    2580 scope.go:117] "RemoveContainer" containerID="ae220ccb8aa246232150ce3b191ff34e7029fce0fdcaff40f550693df013ab90"
Sep 11 20:15:52.035500 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:52.035467    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"]
Sep 11 20:15:52.039538 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:52.039517    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-46da77/autores-model-46da77-kserve-5f56fd66c-m8z9v"]
Sep 11 20:15:52.440904 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:15:52.440876    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d" path="/var/lib/kubelet/pods/dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d/volumes"
Sep 11 20:16:05.092023 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.091985    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"]
Sep 11 20:16:05.092545 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.092373    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e21dd750-305f-48de-84ae-50221476125d" containerName="revoke-keys"
Sep 11 20:16:05.092545 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.092389    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e21dd750-305f-48de-84ae-50221476125d" containerName="revoke-keys"
Sep 11 20:16:05.092545 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.092401    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf" containerName="payload-processing"
Sep 11 20:16:05.092545 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.092408    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf" containerName="payload-processing"
Sep 11 20:16:05.092545 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.092417    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d" containerName="main"
Sep 11 20:16:05.092545 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.092423    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d" containerName="main"
Sep 11 20:16:05.092545 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.092436    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeb00a15-0884-4dee-9ff4-5a444d79a148" containerName="payload-pre-processing"
Sep 11 20:16:05.092545 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.092443    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeb00a15-0884-4dee-9ff4-5a444d79a148" containerName="payload-pre-processing"
Sep 11 20:16:05.092545 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.092451    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85b5ff62-5633-4de3-a7aa-669850db5404" containerName="revoke-keys"
Sep 11 20:16:05.092545 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.092459    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="85b5ff62-5633-4de3-a7aa-669850db5404" containerName="revoke-keys"
Sep 11 20:16:05.092545 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.092550    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="85b5ff62-5633-4de3-a7aa-669850db5404" containerName="revoke-keys"
Sep 11 20:16:05.093146 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.092559    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeb00a15-0884-4dee-9ff4-5a444d79a148" containerName="payload-pre-processing"
Sep 11 20:16:05.093146 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.092566    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="abcbf5fa-adc3-4cdf-b0e6-3b139a3ae9cf" containerName="payload-processing"
Sep 11 20:16:05.093146 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.092571    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd3347ef-ee25-43d7-83e3-d9b3c1dfc51d" containerName="main"
Sep 11 20:16:05.093146 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.092576    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e21dd750-305f-48de-84ae-50221476125d" containerName="revoke-keys"
Sep 11 20:16:05.095686 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.095664    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:16:05.098426 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.098400    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-22fc84\"/\"kube-root-ca.crt\""
Sep 11 20:16:05.099640 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.099617    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-22fc84\"/\"default-dockercfg-gk4zr\""
Sep 11 20:16:05.099743 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.099640    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-22fc84\"/\"openshift-service-ca.crt\""
Sep 11 20:16:05.099743 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.099672    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-22fc84\"/\"premium-w4-22fc84-kserve-self-signed-certs\""
Sep 11 20:16:05.104515 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.104471    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"]
Sep 11 20:16:05.214816 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.214784    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03041350-1de6-4473-ad4c-be9e6ea72bca-tls-certs\") pod \"premium-w4-22fc84-kserve-55f7947b89-htpvs\" (UID: \"03041350-1de6-4473-ad4c-be9e6ea72bca\") " pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:16:05.214816 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.214823    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03041350-1de6-4473-ad4c-be9e6ea72bca-tmp-dir\") pod \"premium-w4-22fc84-kserve-55f7947b89-htpvs\" (UID: \"03041350-1de6-4473-ad4c-be9e6ea72bca\") " pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:16:05.215026 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.214863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03041350-1de6-4473-ad4c-be9e6ea72bca-model-cache\") pod \"premium-w4-22fc84-kserve-55f7947b89-htpvs\" (UID: \"03041350-1de6-4473-ad4c-be9e6ea72bca\") " pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:16:05.215026 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.214889    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03041350-1de6-4473-ad4c-be9e6ea72bca-home\") pod \"premium-w4-22fc84-kserve-55f7947b89-htpvs\" (UID: \"03041350-1de6-4473-ad4c-be9e6ea72bca\") " pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:16:05.215026 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.214904    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03041350-1de6-4473-ad4c-be9e6ea72bca-dshm\") pod \"premium-w4-22fc84-kserve-55f7947b89-htpvs\" (UID: \"03041350-1de6-4473-ad4c-be9e6ea72bca\") " pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:16:05.215026 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.214922    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnnzc\" (UniqueName: \"kubernetes.io/projected/03041350-1de6-4473-ad4c-be9e6ea72bca-kube-api-access-fnnzc\") pod \"premium-w4-22fc84-kserve-55f7947b89-htpvs\" (UID: \"03041350-1de6-4473-ad4c-be9e6ea72bca\") " pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:16:05.315956 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.315913    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03041350-1de6-4473-ad4c-be9e6ea72bca-home\") pod \"premium-w4-22fc84-kserve-55f7947b89-htpvs\" (UID: \"03041350-1de6-4473-ad4c-be9e6ea72bca\") " pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:16:05.315956 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.315956    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03041350-1de6-4473-ad4c-be9e6ea72bca-dshm\") pod \"premium-w4-22fc84-kserve-55f7947b89-htpvs\" (UID: \"03041350-1de6-4473-ad4c-be9e6ea72bca\") " pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:16:05.316179 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.315984    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnnzc\" (UniqueName: \"kubernetes.io/projected/03041350-1de6-4473-ad4c-be9e6ea72bca-kube-api-access-fnnzc\") pod \"premium-w4-22fc84-kserve-55f7947b89-htpvs\" (UID: \"03041350-1de6-4473-ad4c-be9e6ea72bca\") " pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:16:05.316179 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.316084    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03041350-1de6-4473-ad4c-be9e6ea72bca-tls-certs\") pod \"premium-w4-22fc84-kserve-55f7947b89-htpvs\" (UID: \"03041350-1de6-4473-ad4c-be9e6ea72bca\") " pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:16:05.316179 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.316117    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03041350-1de6-4473-ad4c-be9e6ea72bca-tmp-dir\") pod \"premium-w4-22fc84-kserve-55f7947b89-htpvs\" (UID: \"03041350-1de6-4473-ad4c-be9e6ea72bca\") " pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:16:05.316179 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.316143    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03041350-1de6-4473-ad4c-be9e6ea72bca-model-cache\") pod \"premium-w4-22fc84-kserve-55f7947b89-htpvs\" (UID: \"03041350-1de6-4473-ad4c-be9e6ea72bca\") " pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:16:05.316397 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.316347    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03041350-1de6-4473-ad4c-be9e6ea72bca-home\") pod \"premium-w4-22fc84-kserve-55f7947b89-htpvs\" (UID: \"03041350-1de6-4473-ad4c-be9e6ea72bca\") " pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:16:05.316456 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.316428    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03041350-1de6-4473-ad4c-be9e6ea72bca-tmp-dir\") pod \"premium-w4-22fc84-kserve-55f7947b89-htpvs\" (UID: \"03041350-1de6-4473-ad4c-be9e6ea72bca\") " pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:16:05.316542 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.316468    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03041350-1de6-4473-ad4c-be9e6ea72bca-model-cache\") pod \"premium-w4-22fc84-kserve-55f7947b89-htpvs\" (UID: \"03041350-1de6-4473-ad4c-be9e6ea72bca\") " pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:16:05.318244 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.318219    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03041350-1de6-4473-ad4c-be9e6ea72bca-dshm\") pod \"premium-w4-22fc84-kserve-55f7947b89-htpvs\" (UID: \"03041350-1de6-4473-ad4c-be9e6ea72bca\") " pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:16:05.318520 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.318484    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03041350-1de6-4473-ad4c-be9e6ea72bca-tls-certs\") pod \"premium-w4-22fc84-kserve-55f7947b89-htpvs\" (UID: \"03041350-1de6-4473-ad4c-be9e6ea72bca\") " pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:16:05.324031 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.324007    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnnzc\" (UniqueName: \"kubernetes.io/projected/03041350-1de6-4473-ad4c-be9e6ea72bca-kube-api-access-fnnzc\") pod \"premium-w4-22fc84-kserve-55f7947b89-htpvs\" (UID: \"03041350-1de6-4473-ad4c-be9e6ea72bca\") " pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:16:05.406210 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.406121    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:16:05.680849 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:05.680818    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"]
Sep 11 20:16:05.681898 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:16:05.681870    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03041350_1de6_4473_ad4c_be9e6ea72bca.slice/crio-fd87d7361d2b0c4324747fc8074eacb497cca7a35bd02dd88ac33ae7015cb0c1 WatchSource:0}: Error finding container fd87d7361d2b0c4324747fc8074eacb497cca7a35bd02dd88ac33ae7015cb0c1: Status 404 returned error can't find the container with id fd87d7361d2b0c4324747fc8074eacb497cca7a35bd02dd88ac33ae7015cb0c1
Sep 11 20:16:06.098581 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:06.098547    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs" event={"ID":"03041350-1de6-4473-ad4c-be9e6ea72bca","Type":"ContainerStarted","Data":"c32758a94a5fb446c91fbc3bc457ea27a7be613b764df3c4a9b9f4fb7c8b7aca"}
Sep 11 20:16:06.098995 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:06.098590    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs" event={"ID":"03041350-1de6-4473-ad4c-be9e6ea72bca","Type":"ContainerStarted","Data":"fd87d7361d2b0c4324747fc8074eacb497cca7a35bd02dd88ac33ae7015cb0c1"}
Sep 11 20:16:06.115654 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:06.115609    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs" podStartSLOduration=1.115593262 podStartE2EDuration="1.115593262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:16:06.114372681 +0000 UTC m=+936.106534950" watchObservedRunningTime="2026-09-11 20:16:06.115593262 +0000 UTC m=+936.107755531"
Sep 11 20:16:10.808184 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:10.808153    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:16:15.406836 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:15.406793    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:16:15.407213 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:15.406852    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:16:15.419410 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:15.419377    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:16:16.167847 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:16.167811    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:16:29.721133 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:29.721095    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:16:50.578296 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.578259    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"]
Sep 11 20:16:50.583934 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.583904    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:16:50.586981 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.586947    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4f9558\"/\"openshift-service-ca.crt\""
Sep 11 20:16:50.587122 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.587027    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4f9558\"/\"test-model-4f9558-kserve-self-signed-certs\""
Sep 11 20:16:50.587241 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.587219    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4f9558\"/\"kube-root-ca.crt\""
Sep 11 20:16:50.587339 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.587300    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4f9558\"/\"default-dockercfg-ffmwp\""
Sep 11 20:16:50.589751 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.589728    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"]
Sep 11 20:16:50.614417 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.614394    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75d16ee9-8f38-424b-bcd8-224756e9607f-tmp-dir\") pod \"test-model-4f9558-kserve-6696b864c-lxn64\" (UID: \"75d16ee9-8f38-424b-bcd8-224756e9607f\") " pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:16:50.614562 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.614440    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75d16ee9-8f38-424b-bcd8-224756e9607f-home\") pod \"test-model-4f9558-kserve-6696b864c-lxn64\" (UID: \"75d16ee9-8f38-424b-bcd8-224756e9607f\") " pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:16:50.614562 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.614471    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj2b5\" (UniqueName: \"kubernetes.io/projected/75d16ee9-8f38-424b-bcd8-224756e9607f-kube-api-access-zj2b5\") pod \"test-model-4f9558-kserve-6696b864c-lxn64\" (UID: \"75d16ee9-8f38-424b-bcd8-224756e9607f\") " pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:16:50.614648 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.614561    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75d16ee9-8f38-424b-bcd8-224756e9607f-dshm\") pod \"test-model-4f9558-kserve-6696b864c-lxn64\" (UID: \"75d16ee9-8f38-424b-bcd8-224756e9607f\") " pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:16:50.614648 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.614587    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75d16ee9-8f38-424b-bcd8-224756e9607f-tls-certs\") pod \"test-model-4f9558-kserve-6696b864c-lxn64\" (UID: \"75d16ee9-8f38-424b-bcd8-224756e9607f\") " pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:16:50.614648 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.614613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75d16ee9-8f38-424b-bcd8-224756e9607f-model-cache\") pod \"test-model-4f9558-kserve-6696b864c-lxn64\" (UID: \"75d16ee9-8f38-424b-bcd8-224756e9607f\") " pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:16:50.715376 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.715340    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75d16ee9-8f38-424b-bcd8-224756e9607f-tmp-dir\") pod \"test-model-4f9558-kserve-6696b864c-lxn64\" (UID: \"75d16ee9-8f38-424b-bcd8-224756e9607f\") " pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:16:50.715571 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.715398    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75d16ee9-8f38-424b-bcd8-224756e9607f-home\") pod \"test-model-4f9558-kserve-6696b864c-lxn64\" (UID: \"75d16ee9-8f38-424b-bcd8-224756e9607f\") " pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:16:50.715571 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.715430    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj2b5\" (UniqueName: \"kubernetes.io/projected/75d16ee9-8f38-424b-bcd8-224756e9607f-kube-api-access-zj2b5\") pod \"test-model-4f9558-kserve-6696b864c-lxn64\" (UID: \"75d16ee9-8f38-424b-bcd8-224756e9607f\") " pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:16:50.715571 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.715484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75d16ee9-8f38-424b-bcd8-224756e9607f-dshm\") pod \"test-model-4f9558-kserve-6696b864c-lxn64\" (UID: \"75d16ee9-8f38-424b-bcd8-224756e9607f\") " pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:16:50.715571 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.715519    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75d16ee9-8f38-424b-bcd8-224756e9607f-tls-certs\") pod \"test-model-4f9558-kserve-6696b864c-lxn64\" (UID: \"75d16ee9-8f38-424b-bcd8-224756e9607f\") " pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:16:50.715571 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.715546    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75d16ee9-8f38-424b-bcd8-224756e9607f-model-cache\") pod \"test-model-4f9558-kserve-6696b864c-lxn64\" (UID: \"75d16ee9-8f38-424b-bcd8-224756e9607f\") " pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:16:50.715853 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.715833    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75d16ee9-8f38-424b-bcd8-224756e9607f-tmp-dir\") pod \"test-model-4f9558-kserve-6696b864c-lxn64\" (UID: \"75d16ee9-8f38-424b-bcd8-224756e9607f\") " pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:16:50.715892 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.715871    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75d16ee9-8f38-424b-bcd8-224756e9607f-home\") pod \"test-model-4f9558-kserve-6696b864c-lxn64\" (UID: \"75d16ee9-8f38-424b-bcd8-224756e9607f\") " pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:16:50.715930 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.715919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75d16ee9-8f38-424b-bcd8-224756e9607f-model-cache\") pod \"test-model-4f9558-kserve-6696b864c-lxn64\" (UID: \"75d16ee9-8f38-424b-bcd8-224756e9607f\") " pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:16:50.717839 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.717812    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75d16ee9-8f38-424b-bcd8-224756e9607f-dshm\") pod \"test-model-4f9558-kserve-6696b864c-lxn64\" (UID: \"75d16ee9-8f38-424b-bcd8-224756e9607f\") " pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:16:50.718001 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.717984    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75d16ee9-8f38-424b-bcd8-224756e9607f-tls-certs\") pod \"test-model-4f9558-kserve-6696b864c-lxn64\" (UID: \"75d16ee9-8f38-424b-bcd8-224756e9607f\") " pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:16:50.723960 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.723931    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj2b5\" (UniqueName: \"kubernetes.io/projected/75d16ee9-8f38-424b-bcd8-224756e9607f-kube-api-access-zj2b5\") pod \"test-model-4f9558-kserve-6696b864c-lxn64\" (UID: \"75d16ee9-8f38-424b-bcd8-224756e9607f\") " pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:16:50.896253 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:50.896155    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:16:51.171984 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:51.171960    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"]
Sep 11 20:16:51.173993 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:16:51.173959    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75d16ee9_8f38_424b_bcd8_224756e9607f.slice/crio-a9b09cb134dbae8868936b338adf64d0d424a7c3f774f63cb2870d0f0010c7ad WatchSource:0}: Error finding container a9b09cb134dbae8868936b338adf64d0d424a7c3f774f63cb2870d0f0010c7ad: Status 404 returned error can't find the container with id a9b09cb134dbae8868936b338adf64d0d424a7c3f774f63cb2870d0f0010c7ad
Sep 11 20:16:51.357386 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:51.357346    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64" event={"ID":"75d16ee9-8f38-424b-bcd8-224756e9607f","Type":"ContainerStarted","Data":"34c1f73ed9556e9e8e717494f85b309d3a876fa32f36563218d73fe880a71697"}
Sep 11 20:16:51.357386 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:51.357392    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64" event={"ID":"75d16ee9-8f38-424b-bcd8-224756e9607f","Type":"ContainerStarted","Data":"a9b09cb134dbae8868936b338adf64d0d424a7c3f774f63cb2870d0f0010c7ad"}
Sep 11 20:16:51.374599 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:51.374544    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64" podStartSLOduration=1.374525167 podStartE2EDuration="1.374525167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:16:51.373396671 +0000 UTC m=+981.365558952" watchObservedRunningTime="2026-09-11 20:16:51.374525167 +0000 UTC m=+981.366687440"
Sep 11 20:16:58.206599 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:16:58.206520    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:17:00.896960 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:00.896910    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:17:00.896960 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:00.896967    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:17:00.909454 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:00.909431    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:17:01.429692 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:01.429664    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:17:21.715014 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:21.714972    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:17:35.789701 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.789662    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"]
Sep 11 20:17:35.792959 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.792939    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:17:35.797477 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.797453    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-3c2adb\"/\"explicit-model-3c2adb-kserve-self-signed-certs\""
Sep 11 20:17:35.797823 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.797473    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-3c2adb\"/\"kube-root-ca.crt\""
Sep 11 20:17:35.797943 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.797508    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-3c2adb\"/\"default-dockercfg-9p47z\""
Sep 11 20:17:35.798038 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.797506    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-3c2adb\"/\"openshift-service-ca.crt\""
Sep 11 20:17:35.806715 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.806695    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"]
Sep 11 20:17:35.815250 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.815227    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-model-cache\") pod \"explicit-model-3c2adb-kserve-67bfbdb987-5j25m\" (UID: \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\") " pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:17:35.815335 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.815270    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-tls-certs\") pod \"explicit-model-3c2adb-kserve-67bfbdb987-5j25m\" (UID: \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\") " pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:17:35.815387 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.815367    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-tmp-dir\") pod \"explicit-model-3c2adb-kserve-67bfbdb987-5j25m\" (UID: \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\") " pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:17:35.815450 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.815432    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-dshm\") pod \"explicit-model-3c2adb-kserve-67bfbdb987-5j25m\" (UID: \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\") " pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:17:35.815518 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.815480    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmjks\" (UniqueName: \"kubernetes.io/projected/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-kube-api-access-gmjks\") pod \"explicit-model-3c2adb-kserve-67bfbdb987-5j25m\" (UID: \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\") " pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:17:35.815622 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.815538    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-home\") pod \"explicit-model-3c2adb-kserve-67bfbdb987-5j25m\" (UID: \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\") " pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:17:35.916881 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.916844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmjks\" (UniqueName: \"kubernetes.io/projected/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-kube-api-access-gmjks\") pod \"explicit-model-3c2adb-kserve-67bfbdb987-5j25m\" (UID: \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\") " pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:17:35.916881 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.916883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-home\") pod \"explicit-model-3c2adb-kserve-67bfbdb987-5j25m\" (UID: \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\") " pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:17:35.917111 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.916912    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-model-cache\") pod \"explicit-model-3c2adb-kserve-67bfbdb987-5j25m\" (UID: \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\") " pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:17:35.917111 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.916954    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-tls-certs\") pod \"explicit-model-3c2adb-kserve-67bfbdb987-5j25m\" (UID: \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\") " pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:17:35.917111 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.917019    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-tmp-dir\") pod \"explicit-model-3c2adb-kserve-67bfbdb987-5j25m\" (UID: \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\") " pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:17:35.917111 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.917064    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-dshm\") pod \"explicit-model-3c2adb-kserve-67bfbdb987-5j25m\" (UID: \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\") " pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:17:35.917361 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.917340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-home\") pod \"explicit-model-3c2adb-kserve-67bfbdb987-5j25m\" (UID: \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\") " pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:17:35.917408 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.917374    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-model-cache\") pod \"explicit-model-3c2adb-kserve-67bfbdb987-5j25m\" (UID: \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\") " pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:17:35.917408 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.917401    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-tmp-dir\") pod \"explicit-model-3c2adb-kserve-67bfbdb987-5j25m\" (UID: \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\") " pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:17:35.919154 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.919137    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-dshm\") pod \"explicit-model-3c2adb-kserve-67bfbdb987-5j25m\" (UID: \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\") " pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:17:35.919435 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.919420    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-tls-certs\") pod \"explicit-model-3c2adb-kserve-67bfbdb987-5j25m\" (UID: \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\") " pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:17:35.925867 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:35.925845    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmjks\" (UniqueName: \"kubernetes.io/projected/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-kube-api-access-gmjks\") pod \"explicit-model-3c2adb-kserve-67bfbdb987-5j25m\" (UID: \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\") " pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:17:36.104431 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:36.104349    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:17:36.374525 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:36.374484    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"]
Sep 11 20:17:36.376262 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:17:36.376229    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2a5d2af_dc87_4a72_8550_d5b5ff2b4e55.slice/crio-22622ad1d9791e2d24c9247da7bf081568c2f10d6ff5cad9325732cbef1fee86 WatchSource:0}: Error finding container 22622ad1d9791e2d24c9247da7bf081568c2f10d6ff5cad9325732cbef1fee86: Status 404 returned error can't find the container with id 22622ad1d9791e2d24c9247da7bf081568c2f10d6ff5cad9325732cbef1fee86
Sep 11 20:17:36.626267 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:36.626168    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m" event={"ID":"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55","Type":"ContainerStarted","Data":"644d4265083601d85627065b38c2a768f9c59ae403a7d2ee6ccee3258b8e5d94"}
Sep 11 20:17:36.626267 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:36.626218    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m" event={"ID":"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55","Type":"ContainerStarted","Data":"22622ad1d9791e2d24c9247da7bf081568c2f10d6ff5cad9325732cbef1fee86"}
Sep 11 20:17:36.645526 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:36.645461    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m" podStartSLOduration=1.645447634 podStartE2EDuration="1.645447634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:17:36.642783289 +0000 UTC m=+1026.634945571" watchObservedRunningTime="2026-09-11 20:17:36.645447634 +0000 UTC m=+1026.637609903"
Sep 11 20:17:42.408343 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:42.408308    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:17:45.463590 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:45.463555    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6344a120-lsrmr"]
Sep 11 20:17:45.467159 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:45.467143    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6344a120-lsrmr"
Sep 11 20:17:45.469578 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:45.469553    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7dbs5\""
Sep 11 20:17:45.469700 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:45.469557    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:17:45.474318 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:45.474256    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6344a120-lsrmr"]
Sep 11 20:17:45.501111 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:45.501080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d274f46-f8da-42e7-8ba5-41bb070943a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6344a120-lsrmr\" (UID: \"3d274f46-f8da-42e7-8ba5-41bb070943a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6344a120-lsrmr"
Sep 11 20:17:45.601776 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:45.601744    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d274f46-f8da-42e7-8ba5-41bb070943a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6344a120-lsrmr\" (UID: \"3d274f46-f8da-42e7-8ba5-41bb070943a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6344a120-lsrmr"
Sep 11 20:17:45.602393 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:45.602373    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d274f46-f8da-42e7-8ba5-41bb070943a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6344a120-lsrmr\" (UID: \"3d274f46-f8da-42e7-8ba5-41bb070943a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6344a120-lsrmr"
Sep 11 20:17:45.779737 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:45.779647    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6344a120-lsrmr"
Sep 11 20:17:46.051650 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:46.051628    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6344a120-lsrmr"]
Sep 11 20:17:46.053597 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:17:46.053562    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d274f46_f8da_42e7_8ba5_41bb070943a0.slice/crio-b8ca200690756a1b8b029e981b5030b7c2019352074dbc46cde34d02d6512219 WatchSource:0}: Error finding container b8ca200690756a1b8b029e981b5030b7c2019352074dbc46cde34d02d6512219: Status 404 returned error can't find the container with id b8ca200690756a1b8b029e981b5030b7c2019352074dbc46cde34d02d6512219
Sep 11 20:17:46.105245 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:46.105218    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:17:46.105341 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:46.105291    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:17:46.118441 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:46.118418    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:17:46.683656 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:46.683562    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6344a120-lsrmr" event={"ID":"3d274f46-f8da-42e7-8ba5-41bb070943a0","Type":"ContainerStarted","Data":"ad61534e9fb337c8e61ff5a99cac7f35c02d12aecee546bed71f3fc6374a9f5a"}
Sep 11 20:17:46.683656 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:46.683626    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6344a120-lsrmr" event={"ID":"3d274f46-f8da-42e7-8ba5-41bb070943a0","Type":"ContainerStarted","Data":"b8ca200690756a1b8b029e981b5030b7c2019352074dbc46cde34d02d6512219"}
Sep 11 20:17:46.695596 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:46.695569    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:17:46.699997 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:46.699955    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6344a120-lsrmr" podStartSLOduration=1.699941508 podStartE2EDuration="1.699941508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:17:46.697951225 +0000 UTC m=+1036.690113495" watchObservedRunningTime="2026-09-11 20:17:46.699941508 +0000 UTC m=+1036.692103777"
Sep 11 20:17:47.690279 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:47.690240    2580 generic.go:356] "Generic (PLEG): container finished" podID="3d274f46-f8da-42e7-8ba5-41bb070943a0" containerID="ad61534e9fb337c8e61ff5a99cac7f35c02d12aecee546bed71f3fc6374a9f5a" exitCode=0
Sep 11 20:17:47.690279 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:47.690266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6344a120-lsrmr" event={"ID":"3d274f46-f8da-42e7-8ba5-41bb070943a0","Type":"ContainerDied","Data":"ad61534e9fb337c8e61ff5a99cac7f35c02d12aecee546bed71f3fc6374a9f5a"}
Sep 11 20:17:48.830536 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:48.830512    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6344a120-lsrmr"
Sep 11 20:17:48.932640 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:48.932609    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d274f46-f8da-42e7-8ba5-41bb070943a0-maas-api-service-ca\") pod \"3d274f46-f8da-42e7-8ba5-41bb070943a0\" (UID: \"3d274f46-f8da-42e7-8ba5-41bb070943a0\") "
Sep 11 20:17:48.932937 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:48.932915    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d274f46-f8da-42e7-8ba5-41bb070943a0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3d274f46-f8da-42e7-8ba5-41bb070943a0" (UID: "3d274f46-f8da-42e7-8ba5-41bb070943a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:17:48.933007 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:48.932994    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d274f46-f8da-42e7-8ba5-41bb070943a0-maas-api-service-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:17:49.704258 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:49.704227    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6344a120-lsrmr" event={"ID":"3d274f46-f8da-42e7-8ba5-41bb070943a0","Type":"ContainerDied","Data":"b8ca200690756a1b8b029e981b5030b7c2019352074dbc46cde34d02d6512219"}
Sep 11 20:17:49.704427 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:49.704261    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6344a120-lsrmr"
Sep 11 20:17:49.704427 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:49.704283    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8ca200690756a1b8b029e981b5030b7c2019352074dbc46cde34d02d6512219"
Sep 11 20:17:56.205266 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:56.205221    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6344a120-lsrmr"]
Sep 11 20:17:56.207796 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:56.207770    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6344a120-lsrmr"]
Sep 11 20:17:56.441354 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:17:56.441318    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d274f46-f8da-42e7-8ba5-41bb070943a0" path="/var/lib/kubelet/pods/3d274f46-f8da-42e7-8ba5-41bb070943a0/volumes"
Sep 11 20:18:02.296566 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:02.296534    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"]
Sep 11 20:18:02.407941 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:02.407898    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3c2adb-l6cw9"]
Sep 11 20:18:02.408363 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:02.408345    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d274f46-f8da-42e7-8ba5-41bb070943a0" containerName="revoke-keys"
Sep 11 20:18:02.408455 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:02.408365    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d274f46-f8da-42e7-8ba5-41bb070943a0" containerName="revoke-keys"
Sep 11 20:18:02.408535 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:02.408504    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d274f46-f8da-42e7-8ba5-41bb070943a0" containerName="revoke-keys"
Sep 11 20:18:02.411536 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:02.411515    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3c2adb-l6cw9"
Sep 11 20:18:02.414106 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:02.414084    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:18:02.414206 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:02.414138    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7dbs5\""
Sep 11 20:18:02.421771 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:02.421746    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3c2adb-l6cw9"]
Sep 11 20:18:02.550577 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:02.550475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb685f7d-40c7-4bbb-85e5-e518f6f7a939-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-3c2adb-l6cw9\" (UID: \"cb685f7d-40c7-4bbb-85e5-e518f6f7a939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3c2adb-l6cw9"
Sep 11 20:18:02.651705 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:02.651669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb685f7d-40c7-4bbb-85e5-e518f6f7a939-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-3c2adb-l6cw9\" (UID: \"cb685f7d-40c7-4bbb-85e5-e518f6f7a939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3c2adb-l6cw9"
Sep 11 20:18:02.652278 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:02.652257    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb685f7d-40c7-4bbb-85e5-e518f6f7a939-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-3c2adb-l6cw9\" (UID: \"cb685f7d-40c7-4bbb-85e5-e518f6f7a939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3c2adb-l6cw9"
Sep 11 20:18:02.723763 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:02.723729    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3c2adb-l6cw9"
Sep 11 20:18:02.998163 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:02.998139    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3c2adb-l6cw9"]
Sep 11 20:18:03.000972 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:18:03.000936    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb685f7d_40c7_4bbb_85e5_e518f6f7a939.slice/crio-df484c935162f6abf98b7b1e3a36db8e7e5992e2eea7bd1ffa64fd983f2b0a29 WatchSource:0}: Error finding container df484c935162f6abf98b7b1e3a36db8e7e5992e2eea7bd1ffa64fd983f2b0a29: Status 404 returned error can't find the container with id df484c935162f6abf98b7b1e3a36db8e7e5992e2eea7bd1ffa64fd983f2b0a29
Sep 11 20:18:03.792303 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:03.792273    2580 generic.go:356] "Generic (PLEG): container finished" podID="cb685f7d-40c7-4bbb-85e5-e518f6f7a939" containerID="a6d6f52fec18932c8d697e2d6d7a794e73304c2dad159d8a6182cfcb1837f9d9" exitCode=0
Sep 11 20:18:03.792663 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:03.792332    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3c2adb-l6cw9" event={"ID":"cb685f7d-40c7-4bbb-85e5-e518f6f7a939","Type":"ContainerDied","Data":"a6d6f52fec18932c8d697e2d6d7a794e73304c2dad159d8a6182cfcb1837f9d9"}
Sep 11 20:18:03.792663 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:03.792374    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3c2adb-l6cw9" event={"ID":"cb685f7d-40c7-4bbb-85e5-e518f6f7a939","Type":"ContainerStarted","Data":"df484c935162f6abf98b7b1e3a36db8e7e5992e2eea7bd1ffa64fd983f2b0a29"}
Sep 11 20:18:04.941657 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:04.941628    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3c2adb-l6cw9"
Sep 11 20:18:05.073849 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:05.073768    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb685f7d-40c7-4bbb-85e5-e518f6f7a939-maas-api-service-ca\") pod \"cb685f7d-40c7-4bbb-85e5-e518f6f7a939\" (UID: \"cb685f7d-40c7-4bbb-85e5-e518f6f7a939\") "
Sep 11 20:18:05.074080 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:05.074058    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb685f7d-40c7-4bbb-85e5-e518f6f7a939-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cb685f7d-40c7-4bbb-85e5-e518f6f7a939" (UID: "cb685f7d-40c7-4bbb-85e5-e518f6f7a939"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:18:05.174770 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:05.174742    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb685f7d-40c7-4bbb-85e5-e518f6f7a939-maas-api-service-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:18:05.806158 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:05.806126    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3c2adb-l6cw9"
Sep 11 20:18:05.806393 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:05.806161    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3c2adb-l6cw9" event={"ID":"cb685f7d-40c7-4bbb-85e5-e518f6f7a939","Type":"ContainerDied","Data":"df484c935162f6abf98b7b1e3a36db8e7e5992e2eea7bd1ffa64fd983f2b0a29"}
Sep 11 20:18:05.806393 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:05.806210    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df484c935162f6abf98b7b1e3a36db8e7e5992e2eea7bd1ffa64fd983f2b0a29"
Sep 11 20:18:13.264310 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:13.264277    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3c2adb-l6cw9"]
Sep 11 20:18:13.269607 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:13.269580    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-3c2adb-l6cw9"]
Sep 11 20:18:14.441947 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:14.441909    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb685f7d-40c7-4bbb-85e5-e518f6f7a939" path="/var/lib/kubelet/pods/cb685f7d-40c7-4bbb-85e5-e518f6f7a939/volumes"
Sep 11 20:18:15.024741 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:15.024700    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:18:17.315683 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.315625    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m" podUID="e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55" containerName="main" containerID="cri-o://644d4265083601d85627065b38c2a768f9c59ae403a7d2ee6ccee3258b8e5d94" gracePeriod=45
Sep 11 20:18:17.579749 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.579685    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:18:17.687764 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.687736    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-dshm\") pod \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\" (UID: \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\") "
Sep 11 20:18:17.687934 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.687811    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-tls-certs\") pod \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\" (UID: \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\") "
Sep 11 20:18:17.687934 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.687835    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmjks\" (UniqueName: \"kubernetes.io/projected/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-kube-api-access-gmjks\") pod \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\" (UID: \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\") "
Sep 11 20:18:17.687934 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.687872    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-model-cache\") pod \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\" (UID: \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\") "
Sep 11 20:18:17.688073 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.688038    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-home\") pod \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\" (UID: \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\") "
Sep 11 20:18:17.688124 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.688092    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-tmp-dir\") pod \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\" (UID: \"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55\") "
Sep 11 20:18:17.688181 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.688127    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-model-cache" (OuterVolumeSpecName: "model-cache") pod "e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55" (UID: "e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:18:17.688388 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.688331    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-home" (OuterVolumeSpecName: "home") pod "e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55" (UID: "e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:18:17.688522 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.688400    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55" (UID: "e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:18:17.688522 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.688408    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-model-cache\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:18:17.689882 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.689860    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-dshm" (OuterVolumeSpecName: "dshm") pod "e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55" (UID: "e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:18:17.689970 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.689921    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55" (UID: "e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:18:17.689970 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.689954    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-kube-api-access-gmjks" (OuterVolumeSpecName: "kube-api-access-gmjks") pod "e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55" (UID: "e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55"). InnerVolumeSpecName "kube-api-access-gmjks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:18:17.789459 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.789422    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-tls-certs\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:18:17.789459 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.789454    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmjks\" (UniqueName: \"kubernetes.io/projected/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-kube-api-access-gmjks\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:18:17.789669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.789468    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-home\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:18:17.789669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.789482    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-tmp-dir\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:18:17.789669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.789518    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55-dshm\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:18:17.878920 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.878839    2580 generic.go:356] "Generic (PLEG): container finished" podID="e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55" containerID="644d4265083601d85627065b38c2a768f9c59ae403a7d2ee6ccee3258b8e5d94" exitCode=0
Sep 11 20:18:17.878920 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.878897    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"
Sep 11 20:18:17.879133 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.878934    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m" event={"ID":"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55","Type":"ContainerDied","Data":"644d4265083601d85627065b38c2a768f9c59ae403a7d2ee6ccee3258b8e5d94"}
Sep 11 20:18:17.879133 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.878991    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m" event={"ID":"e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55","Type":"ContainerDied","Data":"22622ad1d9791e2d24c9247da7bf081568c2f10d6ff5cad9325732cbef1fee86"}
Sep 11 20:18:17.879133 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.879020    2580 scope.go:117] "RemoveContainer" containerID="644d4265083601d85627065b38c2a768f9c59ae403a7d2ee6ccee3258b8e5d94"
Sep 11 20:18:17.889796 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.889777    2580 scope.go:117] "RemoveContainer" containerID="644d4265083601d85627065b38c2a768f9c59ae403a7d2ee6ccee3258b8e5d94"
Sep 11 20:18:17.890109 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:18:17.890085    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"644d4265083601d85627065b38c2a768f9c59ae403a7d2ee6ccee3258b8e5d94\": container with ID starting with 644d4265083601d85627065b38c2a768f9c59ae403a7d2ee6ccee3258b8e5d94 not found: ID does not exist" containerID="644d4265083601d85627065b38c2a768f9c59ae403a7d2ee6ccee3258b8e5d94"
Sep 11 20:18:17.890186 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.890124    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644d4265083601d85627065b38c2a768f9c59ae403a7d2ee6ccee3258b8e5d94"} err="failed to get container status \"644d4265083601d85627065b38c2a768f9c59ae403a7d2ee6ccee3258b8e5d94\": rpc error: code = NotFound desc = could not find container \"644d4265083601d85627065b38c2a768f9c59ae403a7d2ee6ccee3258b8e5d94\": container with ID starting with 644d4265083601d85627065b38c2a768f9c59ae403a7d2ee6ccee3258b8e5d94 not found: ID does not exist"
Sep 11 20:18:17.910968 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.910932    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"]
Sep 11 20:18:17.913226 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:17.913203    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-3c2adb/explicit-model-3c2adb-kserve-67bfbdb987-5j25m"]
Sep 11 20:18:18.440935 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:18.440906    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55" path="/var/lib/kubelet/pods/e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55/volumes"
Sep 11 20:18:21.579605 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.579567    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"]
Sep 11 20:18:21.580163 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.580139    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55" containerName="main"
Sep 11 20:18:21.580350 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.580325    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55" containerName="main"
Sep 11 20:18:21.580464 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.580376    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb685f7d-40c7-4bbb-85e5-e518f6f7a939" containerName="revoke-keys"
Sep 11 20:18:21.580464 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.580385    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb685f7d-40c7-4bbb-85e5-e518f6f7a939" containerName="revoke-keys"
Sep 11 20:18:21.580630 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.580535    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb685f7d-40c7-4bbb-85e5-e518f6f7a939" containerName="revoke-keys"
Sep 11 20:18:21.580630 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.580551    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2a5d2af-dc87-4a72-8550-d5b5ff2b4e55" containerName="main"
Sep 11 20:18:21.585185 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.585164    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:18:21.587922 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.587902    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-22fc84\"/\"e2e-conflict-b-cb87ecd4-kserve-self-signed-certs\""
Sep 11 20:18:21.594356 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.594333    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"]
Sep 11 20:18:21.726251 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.726215    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf817703-5261-45cf-a8ee-922ca810468e-model-cache\") pod \"e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5\" (UID: \"cf817703-5261-45cf-a8ee-922ca810468e\") " pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:18:21.726416 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.726279    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84b4r\" (UniqueName: \"kubernetes.io/projected/cf817703-5261-45cf-a8ee-922ca810468e-kube-api-access-84b4r\") pod \"e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5\" (UID: \"cf817703-5261-45cf-a8ee-922ca810468e\") " pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:18:21.726416 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.726330    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf817703-5261-45cf-a8ee-922ca810468e-dshm\") pod \"e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5\" (UID: \"cf817703-5261-45cf-a8ee-922ca810468e\") " pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:18:21.726416 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.726349    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf817703-5261-45cf-a8ee-922ca810468e-home\") pod \"e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5\" (UID: \"cf817703-5261-45cf-a8ee-922ca810468e\") " pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:18:21.726416 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.726382    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf817703-5261-45cf-a8ee-922ca810468e-tmp-dir\") pod \"e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5\" (UID: \"cf817703-5261-45cf-a8ee-922ca810468e\") " pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:18:21.726416 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.726400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf817703-5261-45cf-a8ee-922ca810468e-tls-certs\") pod \"e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5\" (UID: \"cf817703-5261-45cf-a8ee-922ca810468e\") " pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:18:21.827255 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.827222    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf817703-5261-45cf-a8ee-922ca810468e-dshm\") pod \"e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5\" (UID: \"cf817703-5261-45cf-a8ee-922ca810468e\") " pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:18:21.827431 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.827262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf817703-5261-45cf-a8ee-922ca810468e-home\") pod \"e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5\" (UID: \"cf817703-5261-45cf-a8ee-922ca810468e\") " pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:18:21.827431 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.827292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf817703-5261-45cf-a8ee-922ca810468e-tmp-dir\") pod \"e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5\" (UID: \"cf817703-5261-45cf-a8ee-922ca810468e\") " pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:18:21.827431 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.827306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf817703-5261-45cf-a8ee-922ca810468e-tls-certs\") pod \"e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5\" (UID: \"cf817703-5261-45cf-a8ee-922ca810468e\") " pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:18:21.827431 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.827343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf817703-5261-45cf-a8ee-922ca810468e-model-cache\") pod \"e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5\" (UID: \"cf817703-5261-45cf-a8ee-922ca810468e\") " pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:18:21.827431 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.827403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84b4r\" (UniqueName: \"kubernetes.io/projected/cf817703-5261-45cf-a8ee-922ca810468e-kube-api-access-84b4r\") pod \"e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5\" (UID: \"cf817703-5261-45cf-a8ee-922ca810468e\") " pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:18:21.827774 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.827716    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf817703-5261-45cf-a8ee-922ca810468e-home\") pod \"e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5\" (UID: \"cf817703-5261-45cf-a8ee-922ca810468e\") " pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:18:21.827774 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.827725    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf817703-5261-45cf-a8ee-922ca810468e-tmp-dir\") pod \"e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5\" (UID: \"cf817703-5261-45cf-a8ee-922ca810468e\") " pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:18:21.827861 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.827771    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf817703-5261-45cf-a8ee-922ca810468e-model-cache\") pod \"e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5\" (UID: \"cf817703-5261-45cf-a8ee-922ca810468e\") " pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:18:21.829586 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.829564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf817703-5261-45cf-a8ee-922ca810468e-dshm\") pod \"e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5\" (UID: \"cf817703-5261-45cf-a8ee-922ca810468e\") " pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:18:21.829817 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.829770    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf817703-5261-45cf-a8ee-922ca810468e-tls-certs\") pod \"e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5\" (UID: \"cf817703-5261-45cf-a8ee-922ca810468e\") " pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:18:21.843898 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.843878    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84b4r\" (UniqueName: \"kubernetes.io/projected/cf817703-5261-45cf-a8ee-922ca810468e-kube-api-access-84b4r\") pod \"e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5\" (UID: \"cf817703-5261-45cf-a8ee-922ca810468e\") " pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:18:21.896361 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:21.896333    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:18:22.168426 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:22.168400    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"]
Sep 11 20:18:22.171086 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:18:22.171061    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf817703_5261_45cf_a8ee_922ca810468e.slice/crio-b0364912347a2240ba830b19b193b124520f21956d64f3e6dd751c6e61523114 WatchSource:0}: Error finding container b0364912347a2240ba830b19b193b124520f21956d64f3e6dd751c6e61523114: Status 404 returned error can't find the container with id b0364912347a2240ba830b19b193b124520f21956d64f3e6dd751c6e61523114
Sep 11 20:18:22.911257 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:22.911219    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5" event={"ID":"cf817703-5261-45cf-a8ee-922ca810468e","Type":"ContainerStarted","Data":"c1c8f953f3c7e40973cb78d715c94ca4b7ab241a9ef2120275d88ff19bdd6d96"}
Sep 11 20:18:22.911727 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:22.911271    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5" event={"ID":"cf817703-5261-45cf-a8ee-922ca810468e","Type":"ContainerStarted","Data":"b0364912347a2240ba830b19b193b124520f21956d64f3e6dd751c6e61523114"}
Sep 11 20:18:22.932656 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:22.932579    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5" podStartSLOduration=1.9325650699999999 podStartE2EDuration="1.93256507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:18:22.931327085 +0000 UTC m=+1072.923489355" watchObservedRunningTime="2026-09-11 20:18:22.93256507 +0000 UTC m=+1072.924727373"
Sep 11 20:18:31.897234 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:31.897198    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:18:31.897234 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:31.897242    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:18:31.910245 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:31.910220    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:18:31.982435 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:31.982411    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:18:54.218228 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:18:54.218191    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:19:23.914858 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:23.914826    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:19:24.426577 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:24.426538    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7553-463fbe"]
Sep 11 20:19:24.430178 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:24.430153    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-463fbe"
Sep 11 20:19:24.433137 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:24.433116    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-94fzf\""
Sep 11 20:19:24.441847 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:24.441828    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7553-463fbe"]
Sep 11 20:19:24.474624 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:24.474592    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp748\" (UniqueName: \"kubernetes.io/projected/75e1ba46-a808-4a56-869c-f744dcb914d5-kube-api-access-wp748\") pod \"test-curl-7553-463fbe\" (UID: \"75e1ba46-a808-4a56-869c-f744dcb914d5\") " pod="opendatahub/test-curl-7553-463fbe"
Sep 11 20:19:24.575810 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:24.575781    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp748\" (UniqueName: \"kubernetes.io/projected/75e1ba46-a808-4a56-869c-f744dcb914d5-kube-api-access-wp748\") pod \"test-curl-7553-463fbe\" (UID: \"75e1ba46-a808-4a56-869c-f744dcb914d5\") " pod="opendatahub/test-curl-7553-463fbe"
Sep 11 20:19:24.584748 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:24.584723    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp748\" (UniqueName: \"kubernetes.io/projected/75e1ba46-a808-4a56-869c-f744dcb914d5-kube-api-access-wp748\") pod \"test-curl-7553-463fbe\" (UID: \"75e1ba46-a808-4a56-869c-f744dcb914d5\") " pod="opendatahub/test-curl-7553-463fbe"
Sep 11 20:19:24.740162 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:24.740138    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-463fbe"
Sep 11 20:19:25.009244 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:25.009218    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7553-463fbe"]
Sep 11 20:19:25.010764 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:19:25.010738    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75e1ba46_a808_4a56_869c_f744dcb914d5.slice/crio-49ef101434daf7de57eb7c327401b86f1769dda21aaa4bd7518b57dfc71f28ef WatchSource:0}: Error finding container 49ef101434daf7de57eb7c327401b86f1769dda21aaa4bd7518b57dfc71f28ef: Status 404 returned error can't find the container with id 49ef101434daf7de57eb7c327401b86f1769dda21aaa4bd7518b57dfc71f28ef
Sep 11 20:19:25.288256 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:25.288161    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7553-463fbe" event={"ID":"75e1ba46-a808-4a56-869c-f744dcb914d5","Type":"ContainerStarted","Data":"303bbfdb633a1163844dd08f9c3322439f678678ebc59db2d3b5c9ca844c0071"}
Sep 11 20:19:25.288256 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:25.288215    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7553-463fbe" event={"ID":"75e1ba46-a808-4a56-869c-f744dcb914d5","Type":"ContainerStarted","Data":"49ef101434daf7de57eb7c327401b86f1769dda21aaa4bd7518b57dfc71f28ef"}
Sep 11 20:19:25.304972 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:25.304925    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7553-463fbe" podStartSLOduration=1.304911556 podStartE2EDuration="1.304911556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:19:25.302646147 +0000 UTC m=+1135.294808418" watchObservedRunningTime="2026-09-11 20:19:25.304911556 +0000 UTC m=+1135.297073826"
Sep 11 20:19:25.862764 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:25.862729    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7553-463fbe"]
Sep 11 20:19:25.866869 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:25.866842    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7553-463fbe"]
Sep 11 20:19:25.980561 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:25.980528    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7553-cbe56d"]
Sep 11 20:19:25.980939 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:25.980927    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e1ba46-a808-4a56-869c-f744dcb914d5" containerName="test-curl-7553-463fbe"
Sep 11 20:19:25.980984 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:25.980941    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e1ba46-a808-4a56-869c-f744dcb914d5" containerName="test-curl-7553-463fbe"
Sep 11 20:19:25.981018 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:25.981003    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="75e1ba46-a808-4a56-869c-f744dcb914d5" containerName="test-curl-7553-463fbe"
Sep 11 20:19:25.984079 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:25.984063    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-cbe56d"
Sep 11 20:19:25.994181 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:25.994157    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7553-cbe56d"]
Sep 11 20:19:26.089181 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:26.089152    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29r5c\" (UniqueName: \"kubernetes.io/projected/18d63601-ecdf-46ad-909a-ebaf52c4cdbb-kube-api-access-29r5c\") pod \"test-curl-7553-cbe56d\" (UID: \"18d63601-ecdf-46ad-909a-ebaf52c4cdbb\") " pod="opendatahub/test-curl-7553-cbe56d"
Sep 11 20:19:26.190330 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:26.190292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29r5c\" (UniqueName: \"kubernetes.io/projected/18d63601-ecdf-46ad-909a-ebaf52c4cdbb-kube-api-access-29r5c\") pod \"test-curl-7553-cbe56d\" (UID: \"18d63601-ecdf-46ad-909a-ebaf52c4cdbb\") " pod="opendatahub/test-curl-7553-cbe56d"
Sep 11 20:19:26.198016 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:26.197990    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29r5c\" (UniqueName: \"kubernetes.io/projected/18d63601-ecdf-46ad-909a-ebaf52c4cdbb-kube-api-access-29r5c\") pod \"test-curl-7553-cbe56d\" (UID: \"18d63601-ecdf-46ad-909a-ebaf52c4cdbb\") " pod="opendatahub/test-curl-7553-cbe56d"
Sep 11 20:19:26.293587 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:26.293477    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-cbe56d"
Sep 11 20:19:26.294924 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:19:26.294900    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7553-463fbe" container="test-curl-7553-463fbe"
Sep 11 20:19:26.563250 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:26.563219    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7553-cbe56d"]
Sep 11 20:19:26.564704 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:19:26.564675    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18d63601_ecdf_46ad_909a_ebaf52c4cdbb.slice/crio-c8b3de749196e5e10bc95118d8cd3170a9aa65dd5dfe5096f3fcc4b33bdcd17d WatchSource:0}: Error finding container c8b3de749196e5e10bc95118d8cd3170a9aa65dd5dfe5096f3fcc4b33bdcd17d: Status 404 returned error can't find the container with id c8b3de749196e5e10bc95118d8cd3170a9aa65dd5dfe5096f3fcc4b33bdcd17d
Sep 11 20:19:27.302711 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:27.302679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7553-cbe56d" event={"ID":"18d63601-ecdf-46ad-909a-ebaf52c4cdbb","Type":"ContainerStarted","Data":"f389a85c9d2b2757ec05695dda2d8d25f6a64e7887baf3aa28fd09c7027fa436"}
Sep 11 20:19:27.302711 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:27.302724    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7553-cbe56d" event={"ID":"18d63601-ecdf-46ad-909a-ebaf52c4cdbb","Type":"ContainerStarted","Data":"c8b3de749196e5e10bc95118d8cd3170a9aa65dd5dfe5096f3fcc4b33bdcd17d"}
Sep 11 20:19:27.303146 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:27.302911    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7553-463fbe" podUID="75e1ba46-a808-4a56-869c-f744dcb914d5" containerName="test-curl-7553-463fbe" containerID="cri-o://303bbfdb633a1163844dd08f9c3322439f678678ebc59db2d3b5c9ca844c0071" gracePeriod=2
Sep 11 20:19:27.318721 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:27.318679    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7553-cbe56d" podStartSLOduration=2.318665747 podStartE2EDuration="2.318665747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:19:27.317304547 +0000 UTC m=+1137.309466818" watchObservedRunningTime="2026-09-11 20:19:27.318665747 +0000 UTC m=+1137.310828058"
Sep 11 20:19:27.863787 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:27.863750    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7553-cbe56d"]
Sep 11 20:19:27.870014 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:27.869984    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7553-cbe56d"]
Sep 11 20:19:27.981332 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:27.981298    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7553-8e68a5"]
Sep 11 20:19:27.981798 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:27.981779    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18d63601-ecdf-46ad-909a-ebaf52c4cdbb" containerName="test-curl-7553-cbe56d"
Sep 11 20:19:27.981903 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:27.981801    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="18d63601-ecdf-46ad-909a-ebaf52c4cdbb" containerName="test-curl-7553-cbe56d"
Sep 11 20:19:27.981958 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:27.981922    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="18d63601-ecdf-46ad-909a-ebaf52c4cdbb" containerName="test-curl-7553-cbe56d"
Sep 11 20:19:27.984993 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:27.984975    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-8e68a5"
Sep 11 20:19:27.989042 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:27.989019    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7553-8e68a5"]
Sep 11 20:19:28.005127 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:28.005095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hg7v\" (UniqueName: \"kubernetes.io/projected/aa4eac42-62bc-45b6-b031-7523860b0088-kube-api-access-2hg7v\") pod \"test-curl-7553-8e68a5\" (UID: \"aa4eac42-62bc-45b6-b031-7523860b0088\") " pod="opendatahub/test-curl-7553-8e68a5"
Sep 11 20:19:28.105852 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:28.105815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hg7v\" (UniqueName: \"kubernetes.io/projected/aa4eac42-62bc-45b6-b031-7523860b0088-kube-api-access-2hg7v\") pod \"test-curl-7553-8e68a5\" (UID: \"aa4eac42-62bc-45b6-b031-7523860b0088\") " pod="opendatahub/test-curl-7553-8e68a5"
Sep 11 20:19:28.113989 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:28.113963    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hg7v\" (UniqueName: \"kubernetes.io/projected/aa4eac42-62bc-45b6-b031-7523860b0088-kube-api-access-2hg7v\") pod \"test-curl-7553-8e68a5\" (UID: \"aa4eac42-62bc-45b6-b031-7523860b0088\") " pod="opendatahub/test-curl-7553-8e68a5"
Sep 11 20:19:28.296394 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:28.296317    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-8e68a5"
Sep 11 20:19:28.309313 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:19:28.309292    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7553-cbe56d" container="test-curl-7553-cbe56d"
Sep 11 20:19:28.569280 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:28.569254    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7553-8e68a5"]
Sep 11 20:19:28.570349 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:19:28.570321    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa4eac42_62bc_45b6_b031_7523860b0088.slice/crio-229c0620d440aaad2bacc2811bcffabd6c8d87fbd10adf749f762f6347d624c8 WatchSource:0}: Error finding container 229c0620d440aaad2bacc2811bcffabd6c8d87fbd10adf749f762f6347d624c8: Status 404 returned error can't find the container with id 229c0620d440aaad2bacc2811bcffabd6c8d87fbd10adf749f762f6347d624c8
Sep 11 20:19:29.321091 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:29.321058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7553-8e68a5" event={"ID":"aa4eac42-62bc-45b6-b031-7523860b0088","Type":"ContainerStarted","Data":"f3354360547bebd8d9290c820695151bd34018e0d7169aec5e4a73f7b8c2ae8b"}
Sep 11 20:19:29.321470 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:29.321116    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7553-8e68a5" event={"ID":"aa4eac42-62bc-45b6-b031-7523860b0088","Type":"ContainerStarted","Data":"229c0620d440aaad2bacc2811bcffabd6c8d87fbd10adf749f762f6347d624c8"}
Sep 11 20:19:29.321470 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:29.321206    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7553-cbe56d" podUID="18d63601-ecdf-46ad-909a-ebaf52c4cdbb" containerName="test-curl-7553-cbe56d" containerID="cri-o://f389a85c9d2b2757ec05695dda2d8d25f6a64e7887baf3aa28fd09c7027fa436" gracePeriod=2
Sep 11 20:19:29.340361 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:29.337987    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7553-8e68a5" podStartSLOduration=2.337968848 podStartE2EDuration="2.337968848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:19:29.334660112 +0000 UTC m=+1139.326822382" watchObservedRunningTime="2026-09-11 20:19:29.337968848 +0000 UTC m=+1139.330131118"
Sep 11 20:19:29.488533 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:29.488509    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-463fbe"
Sep 11 20:19:29.521946 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:29.521921    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wp748\" (UniqueName: \"kubernetes.io/projected/75e1ba46-a808-4a56-869c-f744dcb914d5-kube-api-access-wp748\") pod \"75e1ba46-a808-4a56-869c-f744dcb914d5\" (UID: \"75e1ba46-a808-4a56-869c-f744dcb914d5\") "
Sep 11 20:19:29.523949 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:29.523918    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e1ba46-a808-4a56-869c-f744dcb914d5-kube-api-access-wp748" (OuterVolumeSpecName: "kube-api-access-wp748") pod "75e1ba46-a808-4a56-869c-f744dcb914d5" (UID: "75e1ba46-a808-4a56-869c-f744dcb914d5"). InnerVolumeSpecName "kube-api-access-wp748". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:19:29.623239 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:29.623159    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp748\" (UniqueName: \"kubernetes.io/projected/75e1ba46-a808-4a56-869c-f744dcb914d5-kube-api-access-wp748\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:19:29.887962 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:29.887877    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7553-8e68a5"]
Sep 11 20:19:29.892456 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:29.892429    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7553-8e68a5"]
Sep 11 20:19:29.994569 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:29.994536    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7553-2394fd"]
Sep 11 20:19:29.994949 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:29.994936    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa4eac42-62bc-45b6-b031-7523860b0088" containerName="test-curl-7553-8e68a5"
Sep 11 20:19:29.994999 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:29.994952    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa4eac42-62bc-45b6-b031-7523860b0088" containerName="test-curl-7553-8e68a5"
Sep 11 20:19:29.995037 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:29.995027    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa4eac42-62bc-45b6-b031-7523860b0088" containerName="test-curl-7553-8e68a5"
Sep 11 20:19:29.998049 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:29.998031    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-2394fd"
Sep 11 20:19:30.004194 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:30.004166    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7553-2394fd"]
Sep 11 20:19:30.026412 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:30.026377    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82z2m\" (UniqueName: \"kubernetes.io/projected/1d5747f2-cff7-4780-b780-32c35481fb2a-kube-api-access-82z2m\") pod \"test-curl-7553-2394fd\" (UID: \"1d5747f2-cff7-4780-b780-32c35481fb2a\") " pod="opendatahub/test-curl-7553-2394fd"
Sep 11 20:19:30.127881 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:30.127839    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82z2m\" (UniqueName: \"kubernetes.io/projected/1d5747f2-cff7-4780-b780-32c35481fb2a-kube-api-access-82z2m\") pod \"test-curl-7553-2394fd\" (UID: \"1d5747f2-cff7-4780-b780-32c35481fb2a\") " pod="opendatahub/test-curl-7553-2394fd"
Sep 11 20:19:30.136081 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:30.136048    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82z2m\" (UniqueName: \"kubernetes.io/projected/1d5747f2-cff7-4780-b780-32c35481fb2a-kube-api-access-82z2m\") pod \"test-curl-7553-2394fd\" (UID: \"1d5747f2-cff7-4780-b780-32c35481fb2a\") " pod="opendatahub/test-curl-7553-2394fd"
Sep 11 20:19:30.308640 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:30.308602    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-2394fd"
Sep 11 20:19:30.329410 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:30.329380    2580 generic.go:356] "Generic (PLEG): container finished" podID="75e1ba46-a808-4a56-869c-f744dcb914d5" containerID="303bbfdb633a1163844dd08f9c3322439f678678ebc59db2d3b5c9ca844c0071" exitCode=137
Sep 11 20:19:30.329826 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:30.329438    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-463fbe"
Sep 11 20:19:30.329826 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:30.329511    2580 scope.go:117] "RemoveContainer" containerID="303bbfdb633a1163844dd08f9c3322439f678678ebc59db2d3b5c9ca844c0071"
Sep 11 20:19:30.329940 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:19:30.329830    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7553-8e68a5" container="test-curl-7553-8e68a5"
Sep 11 20:19:30.340969 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:19:30.340942    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7553-463fbe\" is forbidden: User \"system:node:ip-10-0-128-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-209.ec2.internal' and this object" podUID="75e1ba46-a808-4a56-869c-f744dcb914d5" pod="opendatahub/test-curl-7553-463fbe"
Sep 11 20:19:30.341523 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:30.341481    2580 scope.go:117] "RemoveContainer" containerID="303bbfdb633a1163844dd08f9c3322439f678678ebc59db2d3b5c9ca844c0071"
Sep 11 20:19:30.341864 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:19:30.341845    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"303bbfdb633a1163844dd08f9c3322439f678678ebc59db2d3b5c9ca844c0071\": container with ID starting with 303bbfdb633a1163844dd08f9c3322439f678678ebc59db2d3b5c9ca844c0071 not found: ID does not exist" containerID="303bbfdb633a1163844dd08f9c3322439f678678ebc59db2d3b5c9ca844c0071"
Sep 11 20:19:30.341925 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:30.341872    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303bbfdb633a1163844dd08f9c3322439f678678ebc59db2d3b5c9ca844c0071"} err="failed to get container status \"303bbfdb633a1163844dd08f9c3322439f678678ebc59db2d3b5c9ca844c0071\": rpc error: code = NotFound desc = could not find container \"303bbfdb633a1163844dd08f9c3322439f678678ebc59db2d3b5c9ca844c0071\": container with ID starting with 303bbfdb633a1163844dd08f9c3322439f678678ebc59db2d3b5c9ca844c0071 not found: ID does not exist"
Sep 11 20:19:30.439679 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:19:30.439642    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7553-463fbe\" is forbidden: User \"system:node:ip-10-0-128-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-209.ec2.internal' and this object" podUID="75e1ba46-a808-4a56-869c-f744dcb914d5" pod="opendatahub/test-curl-7553-463fbe"
Sep 11 20:19:30.441170 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:30.441150    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75e1ba46-a808-4a56-869c-f744dcb914d5" path="/var/lib/kubelet/pods/75e1ba46-a808-4a56-869c-f744dcb914d5/volumes"
Sep 11 20:19:30.525063 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:30.525033    2580 scope.go:117] "RemoveContainer" containerID="5602f117668c0ba1a0cd9998aafdd8aa58c5ae34a7fc7d935f70ffca8ac79235"
Sep 11 20:19:30.581641 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:30.581617    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7553-2394fd"]
Sep 11 20:19:30.582972 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:19:30.582946    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d5747f2_cff7_4780_b780_32c35481fb2a.slice/crio-16023b0e3cb4801d5b2bef3076b3db424c3682d099345a7a790789238e34d38d WatchSource:0}: Error finding container 16023b0e3cb4801d5b2bef3076b3db424c3682d099345a7a790789238e34d38d: Status 404 returned error can't find the container with id 16023b0e3cb4801d5b2bef3076b3db424c3682d099345a7a790789238e34d38d
Sep 11 20:19:31.337915 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:31.337887    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7553-2394fd" event={"ID":"1d5747f2-cff7-4780-b780-32c35481fb2a","Type":"ContainerStarted","Data":"5b9f9788395ffd1242d349f6f4a31949cbb51cc76cfe03c718f7e3680ddc1181"}
Sep 11 20:19:31.338256 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:31.337927    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7553-2394fd" event={"ID":"1d5747f2-cff7-4780-b780-32c35481fb2a","Type":"ContainerStarted","Data":"16023b0e3cb4801d5b2bef3076b3db424c3682d099345a7a790789238e34d38d"}
Sep 11 20:19:31.338256 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:31.337961    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7553-8e68a5" podUID="aa4eac42-62bc-45b6-b031-7523860b0088" containerName="test-curl-7553-8e68a5" containerID="cri-o://f3354360547bebd8d9290c820695151bd34018e0d7169aec5e4a73f7b8c2ae8b" gracePeriod=2
Sep 11 20:19:31.352385 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:31.352344    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7553-2394fd" podStartSLOduration=2.352330252 podStartE2EDuration="2.352330252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:19:31.351717672 +0000 UTC m=+1141.343879950" watchObservedRunningTime="2026-09-11 20:19:31.352330252 +0000 UTC m=+1141.344492521"
Sep 11 20:19:31.513522 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:31.513487    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-cbe56d"
Sep 11 20:19:31.538707 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:31.538681    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29r5c\" (UniqueName: \"kubernetes.io/projected/18d63601-ecdf-46ad-909a-ebaf52c4cdbb-kube-api-access-29r5c\") pod \"18d63601-ecdf-46ad-909a-ebaf52c4cdbb\" (UID: \"18d63601-ecdf-46ad-909a-ebaf52c4cdbb\") "
Sep 11 20:19:31.540801 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:31.540777    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d63601-ecdf-46ad-909a-ebaf52c4cdbb-kube-api-access-29r5c" (OuterVolumeSpecName: "kube-api-access-29r5c") pod "18d63601-ecdf-46ad-909a-ebaf52c4cdbb" (UID: "18d63601-ecdf-46ad-909a-ebaf52c4cdbb"). InnerVolumeSpecName "kube-api-access-29r5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:19:31.639827 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:31.639738    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29r5c\" (UniqueName: \"kubernetes.io/projected/18d63601-ecdf-46ad-909a-ebaf52c4cdbb-kube-api-access-29r5c\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:19:31.905815 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:31.905726    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7553-2394fd"]
Sep 11 20:19:31.915969 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:31.914150    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7553-2394fd"]
Sep 11 20:19:32.345258 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:32.345222    2580 generic.go:356] "Generic (PLEG): container finished" podID="18d63601-ecdf-46ad-909a-ebaf52c4cdbb" containerID="f389a85c9d2b2757ec05695dda2d8d25f6a64e7887baf3aa28fd09c7027fa436" exitCode=137
Sep 11 20:19:32.345696 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:32.345275    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-cbe56d"
Sep 11 20:19:32.345696 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:32.345313    2580 scope.go:117] "RemoveContainer" containerID="f389a85c9d2b2757ec05695dda2d8d25f6a64e7887baf3aa28fd09c7027fa436"
Sep 11 20:19:32.345696 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:19:32.345665    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7553-2394fd" container="test-curl-7553-2394fd"
Sep 11 20:19:32.354530 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:32.354504    2580 scope.go:117] "RemoveContainer" containerID="f389a85c9d2b2757ec05695dda2d8d25f6a64e7887baf3aa28fd09c7027fa436"
Sep 11 20:19:32.354801 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:19:32.354779    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f389a85c9d2b2757ec05695dda2d8d25f6a64e7887baf3aa28fd09c7027fa436\": container with ID starting with f389a85c9d2b2757ec05695dda2d8d25f6a64e7887baf3aa28fd09c7027fa436 not found: ID does not exist" containerID="f389a85c9d2b2757ec05695dda2d8d25f6a64e7887baf3aa28fd09c7027fa436"
Sep 11 20:19:32.354880 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:32.354806    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f389a85c9d2b2757ec05695dda2d8d25f6a64e7887baf3aa28fd09c7027fa436"} err="failed to get container status \"f389a85c9d2b2757ec05695dda2d8d25f6a64e7887baf3aa28fd09c7027fa436\": rpc error: code = NotFound desc = could not find container \"f389a85c9d2b2757ec05695dda2d8d25f6a64e7887baf3aa28fd09c7027fa436\": container with ID starting with f389a85c9d2b2757ec05695dda2d8d25f6a64e7887baf3aa28fd09c7027fa436 not found: ID does not exist"
Sep 11 20:19:32.355292 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:19:32.355274    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7553-cbe56d\" is forbidden: User \"system:node:ip-10-0-128-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-209.ec2.internal' and this object" podUID="18d63601-ecdf-46ad-909a-ebaf52c4cdbb" pod="opendatahub/test-curl-7553-cbe56d"
Sep 11 20:19:32.441666 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:32.441637    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18d63601-ecdf-46ad-909a-ebaf52c4cdbb" path="/var/lib/kubelet/pods/18d63601-ecdf-46ad-909a-ebaf52c4cdbb/volumes"
Sep 11 20:19:33.353338 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:33.353297    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7553-2394fd" podUID="1d5747f2-cff7-4780-b780-32c35481fb2a" containerName="test-curl-7553-2394fd" containerID="cri-o://5b9f9788395ffd1242d349f6f4a31949cbb51cc76cfe03c718f7e3680ddc1181" gracePeriod=2
Sep 11 20:19:33.519784 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:33.519757    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-8e68a5"
Sep 11 20:19:33.555995 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:33.555963    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hg7v\" (UniqueName: \"kubernetes.io/projected/aa4eac42-62bc-45b6-b031-7523860b0088-kube-api-access-2hg7v\") pod \"aa4eac42-62bc-45b6-b031-7523860b0088\" (UID: \"aa4eac42-62bc-45b6-b031-7523860b0088\") "
Sep 11 20:19:33.557983 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:33.557957    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4eac42-62bc-45b6-b031-7523860b0088-kube-api-access-2hg7v" (OuterVolumeSpecName: "kube-api-access-2hg7v") pod "aa4eac42-62bc-45b6-b031-7523860b0088" (UID: "aa4eac42-62bc-45b6-b031-7523860b0088"). InnerVolumeSpecName "kube-api-access-2hg7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:19:33.657397 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:33.657313    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hg7v\" (UniqueName: \"kubernetes.io/projected/aa4eac42-62bc-45b6-b031-7523860b0088-kube-api-access-2hg7v\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:19:34.361125 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:34.361092    2580 generic.go:356] "Generic (PLEG): container finished" podID="aa4eac42-62bc-45b6-b031-7523860b0088" containerID="f3354360547bebd8d9290c820695151bd34018e0d7169aec5e4a73f7b8c2ae8b" exitCode=137
Sep 11 20:19:34.361584 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:34.361135    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-8e68a5"
Sep 11 20:19:34.361584 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:34.361148    2580 scope.go:117] "RemoveContainer" containerID="f3354360547bebd8d9290c820695151bd34018e0d7169aec5e4a73f7b8c2ae8b"
Sep 11 20:19:34.371007 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:34.370986    2580 scope.go:117] "RemoveContainer" containerID="f3354360547bebd8d9290c820695151bd34018e0d7169aec5e4a73f7b8c2ae8b"
Sep 11 20:19:34.371277 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:19:34.371260    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3354360547bebd8d9290c820695151bd34018e0d7169aec5e4a73f7b8c2ae8b\": container with ID starting with f3354360547bebd8d9290c820695151bd34018e0d7169aec5e4a73f7b8c2ae8b not found: ID does not exist" containerID="f3354360547bebd8d9290c820695151bd34018e0d7169aec5e4a73f7b8c2ae8b"
Sep 11 20:19:34.371350 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:34.371285    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3354360547bebd8d9290c820695151bd34018e0d7169aec5e4a73f7b8c2ae8b"} err="failed to get container status \"f3354360547bebd8d9290c820695151bd34018e0d7169aec5e4a73f7b8c2ae8b\": rpc error: code = NotFound desc = could not find container \"f3354360547bebd8d9290c820695151bd34018e0d7169aec5e4a73f7b8c2ae8b\": container with ID starting with f3354360547bebd8d9290c820695151bd34018e0d7169aec5e4a73f7b8c2ae8b not found: ID does not exist"
Sep 11 20:19:34.371939 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:19:34.371916    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7553-8e68a5\" is forbidden: User \"system:node:ip-10-0-128-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-209.ec2.internal' and this object" podUID="aa4eac42-62bc-45b6-b031-7523860b0088" pod="opendatahub/test-curl-7553-8e68a5"
Sep 11 20:19:34.442012 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:34.441982    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa4eac42-62bc-45b6-b031-7523860b0088" path="/var/lib/kubelet/pods/aa4eac42-62bc-45b6-b031-7523860b0088/volumes"
Sep 11 20:19:35.528470 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:35.528447    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-2394fd"
Sep 11 20:19:35.574308 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:35.574282    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82z2m\" (UniqueName: \"kubernetes.io/projected/1d5747f2-cff7-4780-b780-32c35481fb2a-kube-api-access-82z2m\") pod \"1d5747f2-cff7-4780-b780-32c35481fb2a\" (UID: \"1d5747f2-cff7-4780-b780-32c35481fb2a\") "
Sep 11 20:19:35.576201 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:35.576175    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d5747f2-cff7-4780-b780-32c35481fb2a-kube-api-access-82z2m" (OuterVolumeSpecName: "kube-api-access-82z2m") pod "1d5747f2-cff7-4780-b780-32c35481fb2a" (UID: "1d5747f2-cff7-4780-b780-32c35481fb2a"). InnerVolumeSpecName "kube-api-access-82z2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:19:35.675583 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:35.675512    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82z2m\" (UniqueName: \"kubernetes.io/projected/1d5747f2-cff7-4780-b780-32c35481fb2a-kube-api-access-82z2m\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:19:36.376679 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:36.376646    2580 generic.go:356] "Generic (PLEG): container finished" podID="1d5747f2-cff7-4780-b780-32c35481fb2a" containerID="5b9f9788395ffd1242d349f6f4a31949cbb51cc76cfe03c718f7e3680ddc1181" exitCode=137
Sep 11 20:19:36.376870 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:36.376690    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-2394fd"
Sep 11 20:19:36.376870 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:36.376744    2580 scope.go:117] "RemoveContainer" containerID="5b9f9788395ffd1242d349f6f4a31949cbb51cc76cfe03c718f7e3680ddc1181"
Sep 11 20:19:36.386052 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:36.386034    2580 scope.go:117] "RemoveContainer" containerID="5b9f9788395ffd1242d349f6f4a31949cbb51cc76cfe03c718f7e3680ddc1181"
Sep 11 20:19:36.386329 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:19:36.386312    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b9f9788395ffd1242d349f6f4a31949cbb51cc76cfe03c718f7e3680ddc1181\": container with ID starting with 5b9f9788395ffd1242d349f6f4a31949cbb51cc76cfe03c718f7e3680ddc1181 not found: ID does not exist" containerID="5b9f9788395ffd1242d349f6f4a31949cbb51cc76cfe03c718f7e3680ddc1181"
Sep 11 20:19:36.386385 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:36.386338    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b9f9788395ffd1242d349f6f4a31949cbb51cc76cfe03c718f7e3680ddc1181"} err="failed to get container status \"5b9f9788395ffd1242d349f6f4a31949cbb51cc76cfe03c718f7e3680ddc1181\": rpc error: code = NotFound desc = could not find container \"5b9f9788395ffd1242d349f6f4a31949cbb51cc76cfe03c718f7e3680ddc1181\": container with ID starting with 5b9f9788395ffd1242d349f6f4a31949cbb51cc76cfe03c718f7e3680ddc1181 not found: ID does not exist"
Sep 11 20:19:36.386581 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:19:36.386563    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7553-2394fd\" is forbidden: User \"system:node:ip-10-0-128-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-209.ec2.internal' and this object" podUID="1d5747f2-cff7-4780-b780-32c35481fb2a" pod="opendatahub/test-curl-7553-2394fd"
Sep 11 20:19:36.447310 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:36.447273    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d5747f2-cff7-4780-b780-32c35481fb2a" path="/var/lib/kubelet/pods/1d5747f2-cff7-4780-b780-32c35481fb2a/volumes"
Sep 11 20:19:41.297300 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.297264    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2"]
Sep 11 20:19:41.297859 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.297839    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d5747f2-cff7-4780-b780-32c35481fb2a" containerName="test-curl-7553-2394fd"
Sep 11 20:19:41.297945 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.297862    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d5747f2-cff7-4780-b780-32c35481fb2a" containerName="test-curl-7553-2394fd"
Sep 11 20:19:41.297997 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.297989    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d5747f2-cff7-4780-b780-32c35481fb2a" containerName="test-curl-7553-2394fd"
Sep 11 20:19:41.302283 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.302253    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2"
Sep 11 20:19:41.304832 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.304810    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d11bc81f\""
Sep 11 20:19:41.304934 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.304865    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d11bc81f-dockercfg-cfst2\""
Sep 11 20:19:41.313653 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.313630    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2"]
Sep 11 20:19:41.330727 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.330682    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5"]
Sep 11 20:19:41.334130 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.334114    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5"
Sep 11 20:19:41.344925 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.344903    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5"]
Sep 11 20:19:41.423590 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.423559    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/180469fd-1676-4f54-9b1c-2eba0fc06cdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2\" (UID: \"180469fd-1676-4f54-9b1c-2eba0fc06cdc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2"
Sep 11 20:19:41.423746 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.423595    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7759f6-10df-4e7a-80fe-8e428f4b526a-plugins-config-volume\") pod \"payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5\" (UID: \"fd7759f6-10df-4e7a-80fe-8e428f4b526a\") " pod="openshift-ingress/payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5"
Sep 11 20:19:41.423746 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.423644    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg7fr\" (UniqueName: \"kubernetes.io/projected/180469fd-1676-4f54-9b1c-2eba0fc06cdc-kube-api-access-pg7fr\") pod \"payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2\" (UID: \"180469fd-1676-4f54-9b1c-2eba0fc06cdc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2"
Sep 11 20:19:41.423746 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.423695    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ddr5\" (UniqueName: \"kubernetes.io/projected/fd7759f6-10df-4e7a-80fe-8e428f4b526a-kube-api-access-2ddr5\") pod \"payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5\" (UID: \"fd7759f6-10df-4e7a-80fe-8e428f4b526a\") " pod="openshift-ingress/payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5"
Sep 11 20:19:41.524299 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.524250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ddr5\" (UniqueName: \"kubernetes.io/projected/fd7759f6-10df-4e7a-80fe-8e428f4b526a-kube-api-access-2ddr5\") pod \"payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5\" (UID: \"fd7759f6-10df-4e7a-80fe-8e428f4b526a\") " pod="openshift-ingress/payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5"
Sep 11 20:19:41.524535 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.524473    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/180469fd-1676-4f54-9b1c-2eba0fc06cdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2\" (UID: \"180469fd-1676-4f54-9b1c-2eba0fc06cdc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2"
Sep 11 20:19:41.524683 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.524540    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7759f6-10df-4e7a-80fe-8e428f4b526a-plugins-config-volume\") pod \"payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5\" (UID: \"fd7759f6-10df-4e7a-80fe-8e428f4b526a\") " pod="openshift-ingress/payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5"
Sep 11 20:19:41.524683 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.524593    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg7fr\" (UniqueName: \"kubernetes.io/projected/180469fd-1676-4f54-9b1c-2eba0fc06cdc-kube-api-access-pg7fr\") pod \"payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2\" (UID: \"180469fd-1676-4f54-9b1c-2eba0fc06cdc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2"
Sep 11 20:19:41.525246 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.525227    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7759f6-10df-4e7a-80fe-8e428f4b526a-plugins-config-volume\") pod \"payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5\" (UID: \"fd7759f6-10df-4e7a-80fe-8e428f4b526a\") " pod="openshift-ingress/payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5"
Sep 11 20:19:41.525384 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.525361    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/180469fd-1676-4f54-9b1c-2eba0fc06cdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2\" (UID: \"180469fd-1676-4f54-9b1c-2eba0fc06cdc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2"
Sep 11 20:19:41.533401 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.533374    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ddr5\" (UniqueName: \"kubernetes.io/projected/fd7759f6-10df-4e7a-80fe-8e428f4b526a-kube-api-access-2ddr5\") pod \"payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5\" (UID: \"fd7759f6-10df-4e7a-80fe-8e428f4b526a\") " pod="openshift-ingress/payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5"
Sep 11 20:19:41.534794 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.534772    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg7fr\" (UniqueName: \"kubernetes.io/projected/180469fd-1676-4f54-9b1c-2eba0fc06cdc-kube-api-access-pg7fr\") pod \"payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2\" (UID: \"180469fd-1676-4f54-9b1c-2eba0fc06cdc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2"
Sep 11 20:19:41.612961 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.612874    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2"
Sep 11 20:19:41.644097 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.644062    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5"
Sep 11 20:19:41.893179 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.893047    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2"]
Sep 11 20:19:41.895253 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:19:41.895220    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod180469fd_1676_4f54_9b1c_2eba0fc06cdc.slice/crio-12acd27fb5c6495abf75c94ee487735d38e3223fdee24b7bf044ecc323567d6e WatchSource:0}: Error finding container 12acd27fb5c6495abf75c94ee487735d38e3223fdee24b7bf044ecc323567d6e: Status 404 returned error can't find the container with id 12acd27fb5c6495abf75c94ee487735d38e3223fdee24b7bf044ecc323567d6e
Sep 11 20:19:41.928972 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:41.928948    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5"]
Sep 11 20:19:41.930653 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:19:41.930624    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd7759f6_10df_4e7a_80fe_8e428f4b526a.slice/crio-5721fbb7e2a897088e6067645f785b877bb596805c67d682ade96de13baa1cb7 WatchSource:0}: Error finding container 5721fbb7e2a897088e6067645f785b877bb596805c67d682ade96de13baa1cb7: Status 404 returned error can't find the container with id 5721fbb7e2a897088e6067645f785b877bb596805c67d682ade96de13baa1cb7
Sep 11 20:19:42.417229 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:42.417180    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5" event={"ID":"fd7759f6-10df-4e7a-80fe-8e428f4b526a","Type":"ContainerStarted","Data":"865cc99d1bf1cf63d8a84cddfa1440bdbaf008ce9a1e59a6998680f0fa5896b4"}
Sep 11 20:19:42.417720 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:42.417260    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5"
Sep 11 20:19:42.417720 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:42.417285    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5" event={"ID":"fd7759f6-10df-4e7a-80fe-8e428f4b526a","Type":"ContainerStarted","Data":"5721fbb7e2a897088e6067645f785b877bb596805c67d682ade96de13baa1cb7"}
Sep 11 20:19:42.418803 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:42.418776    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2" event={"ID":"180469fd-1676-4f54-9b1c-2eba0fc06cdc","Type":"ContainerStarted","Data":"2ab8e645b4b5413c00a709083150a8ef4871ee43970fd71d3be547d9612c14e3"}
Sep 11 20:19:42.418915 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:42.418815    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2" event={"ID":"180469fd-1676-4f54-9b1c-2eba0fc06cdc","Type":"ContainerStarted","Data":"12acd27fb5c6495abf75c94ee487735d38e3223fdee24b7bf044ecc323567d6e"}
Sep 11 20:19:42.418915 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:42.418835    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2"
Sep 11 20:19:42.436129 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:42.436079    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5" podStartSLOduration=1.436066052 podStartE2EDuration="1.436066052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:19:42.432782785 +0000 UTC m=+1152.424945078" watchObservedRunningTime="2026-09-11 20:19:42.436066052 +0000 UTC m=+1152.428228339"
Sep 11 20:19:42.448724 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:42.448678    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2" podStartSLOduration=1.4486660709999999 podStartE2EDuration="1.448666071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:19:42.447172203 +0000 UTC m=+1152.439334474" watchObservedRunningTime="2026-09-11 20:19:42.448666071 +0000 UTC m=+1152.440828374"
Sep 11 20:19:45.953847 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:45.953812    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e1dd-7qwfd"]
Sep 11 20:19:45.957584 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:45.957566    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e1dd-7qwfd"
Sep 11 20:19:45.959868 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:45.959846    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7dbs5\""
Sep 11 20:19:45.959947 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:45.959846    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:19:45.975610 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:45.975577    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e1dd-7qwfd"]
Sep 11 20:19:46.069824 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:46.069791    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa005be3-1458-4c14-a974-67e733d4ed26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b3e1dd-7qwfd\" (UID: \"fa005be3-1458-4c14-a974-67e733d4ed26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e1dd-7qwfd"
Sep 11 20:19:46.170532 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:46.170484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa005be3-1458-4c14-a974-67e733d4ed26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b3e1dd-7qwfd\" (UID: \"fa005be3-1458-4c14-a974-67e733d4ed26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e1dd-7qwfd"
Sep 11 20:19:46.171051 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:46.171030    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa005be3-1458-4c14-a974-67e733d4ed26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b3e1dd-7qwfd\" (UID: \"fa005be3-1458-4c14-a974-67e733d4ed26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e1dd-7qwfd"
Sep 11 20:19:46.275116 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:46.275038    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e1dd-7qwfd"
Sep 11 20:19:46.548411 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:46.548380    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e1dd-7qwfd"]
Sep 11 20:19:46.549547 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:19:46.549516    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa005be3_1458_4c14_a974_67e733d4ed26.slice/crio-84c880731fa3f6c5af37e4cc7209389f6fea40b7fd27fb7f4bc9c0a78502dbf2 WatchSource:0}: Error finding container 84c880731fa3f6c5af37e4cc7209389f6fea40b7fd27fb7f4bc9c0a78502dbf2: Status 404 returned error can't find the container with id 84c880731fa3f6c5af37e4cc7209389f6fea40b7fd27fb7f4bc9c0a78502dbf2
Sep 11 20:19:47.451009 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:47.450973    2580 generic.go:356] "Generic (PLEG): container finished" podID="fa005be3-1458-4c14-a974-67e733d4ed26" containerID="3f3047edabf823e2713a74608eb0b17109ca3dc4749ffa773fa9d3a0b4cd7529" exitCode=0
Sep 11 20:19:47.451426 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:47.451053    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e1dd-7qwfd" event={"ID":"fa005be3-1458-4c14-a974-67e733d4ed26","Type":"ContainerDied","Data":"3f3047edabf823e2713a74608eb0b17109ca3dc4749ffa773fa9d3a0b4cd7529"}
Sep 11 20:19:47.451426 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:47.451103    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e1dd-7qwfd" event={"ID":"fa005be3-1458-4c14-a974-67e733d4ed26","Type":"ContainerStarted","Data":"84c880731fa3f6c5af37e4cc7209389f6fea40b7fd27fb7f4bc9c0a78502dbf2"}
Sep 11 20:19:48.593448 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:48.593424    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e1dd-7qwfd"
Sep 11 20:19:48.694911 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:48.694878    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa005be3-1458-4c14-a974-67e733d4ed26-maas-api-service-ca\") pod \"fa005be3-1458-4c14-a974-67e733d4ed26\" (UID: \"fa005be3-1458-4c14-a974-67e733d4ed26\") "
Sep 11 20:19:48.695229 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:48.695206    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa005be3-1458-4c14-a974-67e733d4ed26-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fa005be3-1458-4c14-a974-67e733d4ed26" (UID: "fa005be3-1458-4c14-a974-67e733d4ed26"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:19:48.796602 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:48.796525    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa005be3-1458-4c14-a974-67e733d4ed26-maas-api-service-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:19:49.464423 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:49.464388    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e1dd-7qwfd" event={"ID":"fa005be3-1458-4c14-a974-67e733d4ed26","Type":"ContainerDied","Data":"84c880731fa3f6c5af37e4cc7209389f6fea40b7fd27fb7f4bc9c0a78502dbf2"}
Sep 11 20:19:49.464423 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:49.464413    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e1dd-7qwfd"
Sep 11 20:19:49.464652 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:49.464436    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84c880731fa3f6c5af37e4cc7209389f6fea40b7fd27fb7f4bc9c0a78502dbf2"
Sep 11 20:19:53.426727 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:53.426652    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2"
Sep 11 20:19:53.427069 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:53.426759    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5"
Sep 11 20:19:54.485670 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.485638    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"]
Sep 11 20:19:54.486398 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.486373    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa005be3-1458-4c14-a974-67e733d4ed26" containerName="revoke-keys"
Sep 11 20:19:54.486523 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.486399    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa005be3-1458-4c14-a974-67e733d4ed26" containerName="revoke-keys"
Sep 11 20:19:54.486579 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.486526    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa005be3-1458-4c14-a974-67e733d4ed26" containerName="revoke-keys"
Sep 11 20:19:54.489699 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.489676    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:19:54.492736 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.492716    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-279dba\"/\"openshift-service-ca.crt\""
Sep 11 20:19:54.492842 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.492717    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-279dba\"/\"kube-root-ca.crt\""
Sep 11 20:19:54.493794 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.493772    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-279dba\"/\"default-dockercfg-g8x7m\""
Sep 11 20:19:54.493903 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.493780    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-279dba\"/\"noten-model-279dba-kserve-self-signed-certs\""
Sep 11 20:19:54.503335 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.503281    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"]
Sep 11 20:19:54.652715 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.652679    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-tls-certs\") pod \"noten-model-279dba-kserve-6d7656f755-qqzng\" (UID: \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\") " pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:19:54.652860 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.652724    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-tmp-dir\") pod \"noten-model-279dba-kserve-6d7656f755-qqzng\" (UID: \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\") " pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:19:54.652860 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.652748    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-model-cache\") pod \"noten-model-279dba-kserve-6d7656f755-qqzng\" (UID: \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\") " pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:19:54.652860 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.652829    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62pvz\" (UniqueName: \"kubernetes.io/projected/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-kube-api-access-62pvz\") pod \"noten-model-279dba-kserve-6d7656f755-qqzng\" (UID: \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\") " pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:19:54.652991 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.652901    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-dshm\") pod \"noten-model-279dba-kserve-6d7656f755-qqzng\" (UID: \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\") " pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:19:54.652991 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.652925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-home\") pod \"noten-model-279dba-kserve-6d7656f755-qqzng\" (UID: \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\") " pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:19:54.753676 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.753599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62pvz\" (UniqueName: \"kubernetes.io/projected/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-kube-api-access-62pvz\") pod \"noten-model-279dba-kserve-6d7656f755-qqzng\" (UID: \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\") " pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:19:54.753676 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.753664    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-dshm\") pod \"noten-model-279dba-kserve-6d7656f755-qqzng\" (UID: \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\") " pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:19:54.753851 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.753698    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-home\") pod \"noten-model-279dba-kserve-6d7656f755-qqzng\" (UID: \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\") " pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:19:54.753851 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.753764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-tls-certs\") pod \"noten-model-279dba-kserve-6d7656f755-qqzng\" (UID: \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\") " pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:19:54.753851 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.753807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-tmp-dir\") pod \"noten-model-279dba-kserve-6d7656f755-qqzng\" (UID: \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\") " pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:19:54.753851 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.753840    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-model-cache\") pod \"noten-model-279dba-kserve-6d7656f755-qqzng\" (UID: \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\") " pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:19:54.754154 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.754131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-home\") pod \"noten-model-279dba-kserve-6d7656f755-qqzng\" (UID: \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\") " pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:19:54.754192 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.754161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-model-cache\") pod \"noten-model-279dba-kserve-6d7656f755-qqzng\" (UID: \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\") " pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:19:54.754243 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.754227    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-tmp-dir\") pod \"noten-model-279dba-kserve-6d7656f755-qqzng\" (UID: \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\") " pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:19:54.756056 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.756035    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-dshm\") pod \"noten-model-279dba-kserve-6d7656f755-qqzng\" (UID: \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\") " pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:19:54.756290 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.756272    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-tls-certs\") pod \"noten-model-279dba-kserve-6d7656f755-qqzng\" (UID: \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\") " pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:19:54.762524 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.762482    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62pvz\" (UniqueName: \"kubernetes.io/projected/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-kube-api-access-62pvz\") pod \"noten-model-279dba-kserve-6d7656f755-qqzng\" (UID: \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\") " pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:19:54.801626 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:54.801601    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:19:55.075247 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:55.075224    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"]
Sep 11 20:19:55.077624 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:19:55.077598    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27d886d5_3619_48ac_9a1c_42f49ee7ba9f.slice/crio-ac49fdb542ede4b71d51bfd34683ae002461e90a0dfc5ab0d3748671ed0c7357 WatchSource:0}: Error finding container ac49fdb542ede4b71d51bfd34683ae002461e90a0dfc5ab0d3748671ed0c7357: Status 404 returned error can't find the container with id ac49fdb542ede4b71d51bfd34683ae002461e90a0dfc5ab0d3748671ed0c7357
Sep 11 20:19:55.513659 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:55.513620    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng" event={"ID":"27d886d5-3619-48ac-9a1c-42f49ee7ba9f","Type":"ContainerStarted","Data":"4c02cb48a7db8115ff725f5518fc99234197081535f8fd4410d048d2fc42ef65"}
Sep 11 20:19:55.514039 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:55.513670    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng" event={"ID":"27d886d5-3619-48ac-9a1c-42f49ee7ba9f","Type":"ContainerStarted","Data":"ac49fdb542ede4b71d51bfd34683ae002461e90a0dfc5ab0d3748671ed0c7357"}
Sep 11 20:19:55.530239 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:55.530187    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng" podStartSLOduration=1.5301707850000001 podStartE2EDuration="1.530170785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:19:55.528994407 +0000 UTC m=+1165.521156689" watchObservedRunningTime="2026-09-11 20:19:55.530170785 +0000 UTC m=+1165.522333054"
Sep 11 20:19:58.379275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:58.379235    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e1dd-29819295-rj4cn"]
Sep 11 20:19:58.381956 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:58.381929    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-b3e1dd-29819295-rj4cn"]
Sep 11 20:19:58.402772 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:58.402743    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx"]
Sep 11 20:19:58.403045 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:58.402992    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx" podUID="a0588500-5e82-4cfe-9a88-b5795b7b03a5" containerName="payload-processing" containerID="cri-o://c087ffb4b2fd2cbe5ae1d4c419cb482b21c942d64fdd667d1af7bd971de851ee" gracePeriod=30
Sep 11 20:19:58.441768 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:58.441740    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c995bc8-6b06-4897-a745-7e19aa68441b" path="/var/lib/kubelet/pods/2c995bc8-6b06-4897-a745-7e19aa68441b/volumes"
Sep 11 20:19:58.537636 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:58.537604    2580 generic.go:356] "Generic (PLEG): container finished" podID="a0588500-5e82-4cfe-9a88-b5795b7b03a5" containerID="c087ffb4b2fd2cbe5ae1d4c419cb482b21c942d64fdd667d1af7bd971de851ee" exitCode=0
Sep 11 20:19:58.537815 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:58.537675    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx" event={"ID":"a0588500-5e82-4cfe-9a88-b5795b7b03a5","Type":"ContainerDied","Data":"c087ffb4b2fd2cbe5ae1d4c419cb482b21c942d64fdd667d1af7bd971de851ee"}
Sep 11 20:19:58.613136 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:58.613108    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e1dd-7qwfd"]
Sep 11 20:19:58.617179 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:58.617153    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e1dd-7qwfd"]
Sep 11 20:19:58.659030 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:58.659006    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx"
Sep 11 20:19:58.795511 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:58.795469    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxzlw\" (UniqueName: \"kubernetes.io/projected/a0588500-5e82-4cfe-9a88-b5795b7b03a5-kube-api-access-lxzlw\") pod \"a0588500-5e82-4cfe-9a88-b5795b7b03a5\" (UID: \"a0588500-5e82-4cfe-9a88-b5795b7b03a5\") "
Sep 11 20:19:58.795679 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:58.795571    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0588500-5e82-4cfe-9a88-b5795b7b03a5-plugins-config-volume\") pod \"a0588500-5e82-4cfe-9a88-b5795b7b03a5\" (UID: \"a0588500-5e82-4cfe-9a88-b5795b7b03a5\") "
Sep 11 20:19:58.795964 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:58.795940    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0588500-5e82-4cfe-9a88-b5795b7b03a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a0588500-5e82-4cfe-9a88-b5795b7b03a5" (UID: "a0588500-5e82-4cfe-9a88-b5795b7b03a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:19:58.797944 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:58.797923    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0588500-5e82-4cfe-9a88-b5795b7b03a5-kube-api-access-lxzlw" (OuterVolumeSpecName: "kube-api-access-lxzlw") pod "a0588500-5e82-4cfe-9a88-b5795b7b03a5" (UID: "a0588500-5e82-4cfe-9a88-b5795b7b03a5"). InnerVolumeSpecName "kube-api-access-lxzlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:19:58.896510 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:58.896432    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxzlw\" (UniqueName: \"kubernetes.io/projected/a0588500-5e82-4cfe-9a88-b5795b7b03a5-kube-api-access-lxzlw\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:19:58.896510 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:58.896456    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0588500-5e82-4cfe-9a88-b5795b7b03a5-plugins-config-volume\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:19:59.545391 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:59.545358    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx"
Sep 11 20:19:59.545391 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:59.545384    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx" event={"ID":"a0588500-5e82-4cfe-9a88-b5795b7b03a5","Type":"ContainerDied","Data":"6c31c6e21d223154d35cd5b7a5017a6fdb54c05ea6bc5ce4fb36baa55a73e1d6"}
Sep 11 20:19:59.545904 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:59.545448    2580 scope.go:117] "RemoveContainer" containerID="c087ffb4b2fd2cbe5ae1d4c419cb482b21c942d64fdd667d1af7bd971de851ee"
Sep 11 20:19:59.575991 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:59.575937    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx"]
Sep 11 20:19:59.580918 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:19:59.580893    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b3e1dd-db988d4cf-hvfzx"]
Sep 11 20:20:00.443852 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:00.443816    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0588500-5e82-4cfe-9a88-b5795b7b03a5" path="/var/lib/kubelet/pods/a0588500-5e82-4cfe-9a88-b5795b7b03a5/volumes"
Sep 11 20:20:00.444170 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:00.444157    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa005be3-1458-4c14-a974-67e733d4ed26" path="/var/lib/kubelet/pods/fa005be3-1458-4c14-a974-67e733d4ed26/volumes"
Sep 11 20:20:02.510852 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:02.510818    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:20:03.917720 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:03.917683    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"]
Sep 11 20:20:04.801928 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:04.801883    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:20:04.802112 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:04.801951    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:20:04.814597 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:04.814563    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:20:05.596742 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:05.596711    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:20:18.936950 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:18.936880    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9" podUID="048e46f1-7821-4ecd-8cb4-93b091672c59" containerName="main" containerID="cri-o://7dbbde7a5515cb64db2e3406be19c023837c3bdc1de3ac0bec1e2df37f698a06" gracePeriod=45
Sep 11 20:20:19.189145 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.189085    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:20:19.283275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.283241    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnfjw\" (UniqueName: \"kubernetes.io/projected/048e46f1-7821-4ecd-8cb4-93b091672c59-kube-api-access-gnfjw\") pod \"048e46f1-7821-4ecd-8cb4-93b091672c59\" (UID: \"048e46f1-7821-4ecd-8cb4-93b091672c59\") "
Sep 11 20:20:19.283434 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.283303    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/048e46f1-7821-4ecd-8cb4-93b091672c59-dshm\") pod \"048e46f1-7821-4ecd-8cb4-93b091672c59\" (UID: \"048e46f1-7821-4ecd-8cb4-93b091672c59\") "
Sep 11 20:20:19.283434 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.283352    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/048e46f1-7821-4ecd-8cb4-93b091672c59-model-cache\") pod \"048e46f1-7821-4ecd-8cb4-93b091672c59\" (UID: \"048e46f1-7821-4ecd-8cb4-93b091672c59\") "
Sep 11 20:20:19.283434 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.283371    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/048e46f1-7821-4ecd-8cb4-93b091672c59-home\") pod \"048e46f1-7821-4ecd-8cb4-93b091672c59\" (UID: \"048e46f1-7821-4ecd-8cb4-93b091672c59\") "
Sep 11 20:20:19.283434 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.283407    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/048e46f1-7821-4ecd-8cb4-93b091672c59-tmp-dir\") pod \"048e46f1-7821-4ecd-8cb4-93b091672c59\" (UID: \"048e46f1-7821-4ecd-8cb4-93b091672c59\") "
Sep 11 20:20:19.283652 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.283457    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/048e46f1-7821-4ecd-8cb4-93b091672c59-tls-certs\") pod \"048e46f1-7821-4ecd-8cb4-93b091672c59\" (UID: \"048e46f1-7821-4ecd-8cb4-93b091672c59\") "
Sep 11 20:20:19.283652 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.283624    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048e46f1-7821-4ecd-8cb4-93b091672c59-model-cache" (OuterVolumeSpecName: "model-cache") pod "048e46f1-7821-4ecd-8cb4-93b091672c59" (UID: "048e46f1-7821-4ecd-8cb4-93b091672c59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:20:19.283761 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.283636    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048e46f1-7821-4ecd-8cb4-93b091672c59-home" (OuterVolumeSpecName: "home") pod "048e46f1-7821-4ecd-8cb4-93b091672c59" (UID: "048e46f1-7821-4ecd-8cb4-93b091672c59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:20:19.283761 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.283734    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048e46f1-7821-4ecd-8cb4-93b091672c59-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "048e46f1-7821-4ecd-8cb4-93b091672c59" (UID: "048e46f1-7821-4ecd-8cb4-93b091672c59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:20:19.283864 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.283794    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/048e46f1-7821-4ecd-8cb4-93b091672c59-model-cache\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:20:19.283864 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.283812    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/048e46f1-7821-4ecd-8cb4-93b091672c59-home\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:20:19.285542 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.285519    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048e46f1-7821-4ecd-8cb4-93b091672c59-kube-api-access-gnfjw" (OuterVolumeSpecName: "kube-api-access-gnfjw") pod "048e46f1-7821-4ecd-8cb4-93b091672c59" (UID: "048e46f1-7821-4ecd-8cb4-93b091672c59"). InnerVolumeSpecName "kube-api-access-gnfjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:20:19.285605 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.285587    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048e46f1-7821-4ecd-8cb4-93b091672c59-dshm" (OuterVolumeSpecName: "dshm") pod "048e46f1-7821-4ecd-8cb4-93b091672c59" (UID: "048e46f1-7821-4ecd-8cb4-93b091672c59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:20:19.285605 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.285585    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/048e46f1-7821-4ecd-8cb4-93b091672c59-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "048e46f1-7821-4ecd-8cb4-93b091672c59" (UID: "048e46f1-7821-4ecd-8cb4-93b091672c59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:20:19.384414 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.384374    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/048e46f1-7821-4ecd-8cb4-93b091672c59-dshm\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:20:19.384414 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.384407    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/048e46f1-7821-4ecd-8cb4-93b091672c59-tmp-dir\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:20:19.384414 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.384417    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/048e46f1-7821-4ecd-8cb4-93b091672c59-tls-certs\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:20:19.384414 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.384425    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnfjw\" (UniqueName: \"kubernetes.io/projected/048e46f1-7821-4ecd-8cb4-93b091672c59-kube-api-access-gnfjw\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:20:19.676769 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.676672    2580 generic.go:356] "Generic (PLEG): container finished" podID="048e46f1-7821-4ecd-8cb4-93b091672c59" containerID="7dbbde7a5515cb64db2e3406be19c023837c3bdc1de3ac0bec1e2df37f698a06" exitCode=0
Sep 11 20:20:19.676769 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.676720    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"
Sep 11 20:20:19.677015 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.676763    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9" event={"ID":"048e46f1-7821-4ecd-8cb4-93b091672c59","Type":"ContainerDied","Data":"7dbbde7a5515cb64db2e3406be19c023837c3bdc1de3ac0bec1e2df37f698a06"}
Sep 11 20:20:19.677015 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.676818    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9" event={"ID":"048e46f1-7821-4ecd-8cb4-93b091672c59","Type":"ContainerDied","Data":"c985846dac843cf1a9a5f6b0de813c0b9c4e4b13fbcbe3bcafa936af2c83c4ea"}
Sep 11 20:20:19.677015 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.676841    2580 scope.go:117] "RemoveContainer" containerID="7dbbde7a5515cb64db2e3406be19c023837c3bdc1de3ac0bec1e2df37f698a06"
Sep 11 20:20:19.686909 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.686888    2580 scope.go:117] "RemoveContainer" containerID="7dbbde7a5515cb64db2e3406be19c023837c3bdc1de3ac0bec1e2df37f698a06"
Sep 11 20:20:19.687196 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:20:19.687177    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dbbde7a5515cb64db2e3406be19c023837c3bdc1de3ac0bec1e2df37f698a06\": container with ID starting with 7dbbde7a5515cb64db2e3406be19c023837c3bdc1de3ac0bec1e2df37f698a06 not found: ID does not exist" containerID="7dbbde7a5515cb64db2e3406be19c023837c3bdc1de3ac0bec1e2df37f698a06"
Sep 11 20:20:19.687272 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.687205    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dbbde7a5515cb64db2e3406be19c023837c3bdc1de3ac0bec1e2df37f698a06"} err="failed to get container status \"7dbbde7a5515cb64db2e3406be19c023837c3bdc1de3ac0bec1e2df37f698a06\": rpc error: code = NotFound desc = could not find container \"7dbbde7a5515cb64db2e3406be19c023837c3bdc1de3ac0bec1e2df37f698a06\": container with ID starting with 7dbbde7a5515cb64db2e3406be19c023837c3bdc1de3ac0bec1e2df37f698a06 not found: ID does not exist"
Sep 11 20:20:19.700590 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.700564    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"]
Sep 11 20:20:19.704569 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:19.704548    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b3e1dd/test-model-b3e1dd-kserve-79d5488dcf-lk6s9"]
Sep 11 20:20:20.441771 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:20.441735    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="048e46f1-7821-4ecd-8cb4-93b091672c59" path="/var/lib/kubelet/pods/048e46f1-7821-4ecd-8cb4-93b091672c59/volumes"
Sep 11 20:20:30.551280 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:30.551245    2580 scope.go:117] "RemoveContainer" containerID="39df5188ec8960b0f6e73dbbfc9108bbde89b6a62fe6fc9c13887739e15d5c02"
Sep 11 20:20:31.362050 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:31.362012    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"]
Sep 11 20:20:33.709196 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:33.709155    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:20:38.695165 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.695120    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"]
Sep 11 20:20:38.696132 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.696106    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0588500-5e82-4cfe-9a88-b5795b7b03a5" containerName="payload-processing"
Sep 11 20:20:38.696214 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.696136    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0588500-5e82-4cfe-9a88-b5795b7b03a5" containerName="payload-processing"
Sep 11 20:20:38.696214 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.696194    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="048e46f1-7821-4ecd-8cb4-93b091672c59" containerName="main"
Sep 11 20:20:38.696214 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.696204    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="048e46f1-7821-4ecd-8cb4-93b091672c59" containerName="main"
Sep 11 20:20:38.696586 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.696436    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0588500-5e82-4cfe-9a88-b5795b7b03a5" containerName="payload-processing"
Sep 11 20:20:38.696586 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.696464    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="048e46f1-7821-4ecd-8cb4-93b091672c59" containerName="main"
Sep 11 20:20:38.715427 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.708737    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:20:38.715427 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.708802    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"]
Sep 11 20:20:38.715733 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.715706    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9d9abb\"/\"distinct-w0-9d9abb-kserve-self-signed-certs\""
Sep 11 20:20:38.867219 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.867184    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wsxs\" (UniqueName: \"kubernetes.io/projected/a81a2439-0fc8-4113-88f1-b5903a79cba8-kube-api-access-6wsxs\") pod \"distinct-w0-9d9abb-kserve-65b94b548d-ft6bg\" (UID: \"a81a2439-0fc8-4113-88f1-b5903a79cba8\") " pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:20:38.867219 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.867221    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a81a2439-0fc8-4113-88f1-b5903a79cba8-tls-certs\") pod \"distinct-w0-9d9abb-kserve-65b94b548d-ft6bg\" (UID: \"a81a2439-0fc8-4113-88f1-b5903a79cba8\") " pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:20:38.867421 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.867289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a81a2439-0fc8-4113-88f1-b5903a79cba8-tmp-dir\") pod \"distinct-w0-9d9abb-kserve-65b94b548d-ft6bg\" (UID: \"a81a2439-0fc8-4113-88f1-b5903a79cba8\") " pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:20:38.867421 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.867378    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a81a2439-0fc8-4113-88f1-b5903a79cba8-model-cache\") pod \"distinct-w0-9d9abb-kserve-65b94b548d-ft6bg\" (UID: \"a81a2439-0fc8-4113-88f1-b5903a79cba8\") " pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:20:38.867421 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.867406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a81a2439-0fc8-4113-88f1-b5903a79cba8-home\") pod \"distinct-w0-9d9abb-kserve-65b94b548d-ft6bg\" (UID: \"a81a2439-0fc8-4113-88f1-b5903a79cba8\") " pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:20:38.867554 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.867426    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a81a2439-0fc8-4113-88f1-b5903a79cba8-dshm\") pod \"distinct-w0-9d9abb-kserve-65b94b548d-ft6bg\" (UID: \"a81a2439-0fc8-4113-88f1-b5903a79cba8\") " pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:20:38.968953 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.968843    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a81a2439-0fc8-4113-88f1-b5903a79cba8-model-cache\") pod \"distinct-w0-9d9abb-kserve-65b94b548d-ft6bg\" (UID: \"a81a2439-0fc8-4113-88f1-b5903a79cba8\") " pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:20:38.968953 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.968897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a81a2439-0fc8-4113-88f1-b5903a79cba8-home\") pod \"distinct-w0-9d9abb-kserve-65b94b548d-ft6bg\" (UID: \"a81a2439-0fc8-4113-88f1-b5903a79cba8\") " pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:20:38.968953 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.968920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a81a2439-0fc8-4113-88f1-b5903a79cba8-dshm\") pod \"distinct-w0-9d9abb-kserve-65b94b548d-ft6bg\" (UID: \"a81a2439-0fc8-4113-88f1-b5903a79cba8\") " pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:20:38.969205 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.968973    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wsxs\" (UniqueName: \"kubernetes.io/projected/a81a2439-0fc8-4113-88f1-b5903a79cba8-kube-api-access-6wsxs\") pod \"distinct-w0-9d9abb-kserve-65b94b548d-ft6bg\" (UID: \"a81a2439-0fc8-4113-88f1-b5903a79cba8\") " pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:20:38.969205 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.968992    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a81a2439-0fc8-4113-88f1-b5903a79cba8-tls-certs\") pod \"distinct-w0-9d9abb-kserve-65b94b548d-ft6bg\" (UID: \"a81a2439-0fc8-4113-88f1-b5903a79cba8\") " pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:20:38.969205 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.969045    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a81a2439-0fc8-4113-88f1-b5903a79cba8-tmp-dir\") pod \"distinct-w0-9d9abb-kserve-65b94b548d-ft6bg\" (UID: \"a81a2439-0fc8-4113-88f1-b5903a79cba8\") " pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:20:38.969333 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.969296    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a81a2439-0fc8-4113-88f1-b5903a79cba8-model-cache\") pod \"distinct-w0-9d9abb-kserve-65b94b548d-ft6bg\" (UID: \"a81a2439-0fc8-4113-88f1-b5903a79cba8\") " pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:20:38.969442 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.969329    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a81a2439-0fc8-4113-88f1-b5903a79cba8-home\") pod \"distinct-w0-9d9abb-kserve-65b94b548d-ft6bg\" (UID: \"a81a2439-0fc8-4113-88f1-b5903a79cba8\") " pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:20:38.969442 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.969390    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a81a2439-0fc8-4113-88f1-b5903a79cba8-tmp-dir\") pod \"distinct-w0-9d9abb-kserve-65b94b548d-ft6bg\" (UID: \"a81a2439-0fc8-4113-88f1-b5903a79cba8\") " pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:20:38.971293 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.971264    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a81a2439-0fc8-4113-88f1-b5903a79cba8-dshm\") pod \"distinct-w0-9d9abb-kserve-65b94b548d-ft6bg\" (UID: \"a81a2439-0fc8-4113-88f1-b5903a79cba8\") " pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:20:38.971445 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.971427    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a81a2439-0fc8-4113-88f1-b5903a79cba8-tls-certs\") pod \"distinct-w0-9d9abb-kserve-65b94b548d-ft6bg\" (UID: \"a81a2439-0fc8-4113-88f1-b5903a79cba8\") " pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:20:38.978002 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:38.977979    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wsxs\" (UniqueName: \"kubernetes.io/projected/a81a2439-0fc8-4113-88f1-b5903a79cba8-kube-api-access-6wsxs\") pod \"distinct-w0-9d9abb-kserve-65b94b548d-ft6bg\" (UID: \"a81a2439-0fc8-4113-88f1-b5903a79cba8\") " pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:20:39.024844 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:39.024813    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:20:39.298543 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:39.298518    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"]
Sep 11 20:20:39.300466 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:20:39.300440    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda81a2439_0fc8_4113_88f1_b5903a79cba8.slice/crio-55d3b649ec6e0cf3ed08a1575ba2ceb88e1b97762090af2b9c87ef281812a4b1 WatchSource:0}: Error finding container 55d3b649ec6e0cf3ed08a1575ba2ceb88e1b97762090af2b9c87ef281812a4b1: Status 404 returned error can't find the container with id 55d3b649ec6e0cf3ed08a1575ba2ceb88e1b97762090af2b9c87ef281812a4b1
Sep 11 20:20:39.797699 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:39.797655    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg" event={"ID":"a81a2439-0fc8-4113-88f1-b5903a79cba8","Type":"ContainerStarted","Data":"aba30f8251f7185ad501edfa4da44786afb1e8b6fddd434c14a9ce0d04091d70"}
Sep 11 20:20:39.798158 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:39.797712    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg" event={"ID":"a81a2439-0fc8-4113-88f1-b5903a79cba8","Type":"ContainerStarted","Data":"55d3b649ec6e0cf3ed08a1575ba2ceb88e1b97762090af2b9c87ef281812a4b1"}
Sep 11 20:20:39.818993 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:39.818943    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg" podStartSLOduration=1.818926366 podStartE2EDuration="1.818926366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:20:39.814810877 +0000 UTC m=+1209.806973147" watchObservedRunningTime="2026-09-11 20:20:39.818926366 +0000 UTC m=+1209.811088636"
Sep 11 20:20:44.960170 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:44.960132    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"]
Sep 11 20:20:46.381440 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.381375    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng" podUID="27d886d5-3619-48ac-9a1c-42f49ee7ba9f" containerName="main" containerID="cri-o://4c02cb48a7db8115ff725f5518fc99234197081535f8fd4410d048d2fc42ef65" gracePeriod=45
Sep 11 20:20:46.646002 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.645947    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:20:46.753114 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.753086    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-home\") pod \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\" (UID: \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\") "
Sep 11 20:20:46.753303 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.753196    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-tls-certs\") pod \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\" (UID: \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\") "
Sep 11 20:20:46.753303 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.753223    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-tmp-dir\") pod \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\" (UID: \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\") "
Sep 11 20:20:46.753303 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.753263    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62pvz\" (UniqueName: \"kubernetes.io/projected/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-kube-api-access-62pvz\") pod \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\" (UID: \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\") "
Sep 11 20:20:46.753303 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.753292    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-dshm\") pod \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\" (UID: \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\") "
Sep 11 20:20:46.753592 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.753376    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-home" (OuterVolumeSpecName: "home") pod "27d886d5-3619-48ac-9a1c-42f49ee7ba9f" (UID: "27d886d5-3619-48ac-9a1c-42f49ee7ba9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:20:46.753592 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.753430    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-model-cache\") pod \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\" (UID: \"27d886d5-3619-48ac-9a1c-42f49ee7ba9f\") "
Sep 11 20:20:46.753592 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.753509    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "27d886d5-3619-48ac-9a1c-42f49ee7ba9f" (UID: "27d886d5-3619-48ac-9a1c-42f49ee7ba9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:20:46.753731 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.753685    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-model-cache" (OuterVolumeSpecName: "model-cache") pod "27d886d5-3619-48ac-9a1c-42f49ee7ba9f" (UID: "27d886d5-3619-48ac-9a1c-42f49ee7ba9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:20:46.753908 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.753885    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-home\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:20:46.754035 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.753911    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-tmp-dir\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:20:46.754035 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.753926    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-model-cache\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:20:46.755452 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.755428    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27d886d5-3619-48ac-9a1c-42f49ee7ba9f" (UID: "27d886d5-3619-48ac-9a1c-42f49ee7ba9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:20:46.755562 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.755484    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-dshm" (OuterVolumeSpecName: "dshm") pod "27d886d5-3619-48ac-9a1c-42f49ee7ba9f" (UID: "27d886d5-3619-48ac-9a1c-42f49ee7ba9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:20:46.755562 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.755488    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-kube-api-access-62pvz" (OuterVolumeSpecName: "kube-api-access-62pvz") pod "27d886d5-3619-48ac-9a1c-42f49ee7ba9f" (UID: "27d886d5-3619-48ac-9a1c-42f49ee7ba9f"). InnerVolumeSpecName "kube-api-access-62pvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:20:46.853048 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.853013    2580 generic.go:356] "Generic (PLEG): container finished" podID="27d886d5-3619-48ac-9a1c-42f49ee7ba9f" containerID="4c02cb48a7db8115ff725f5518fc99234197081535f8fd4410d048d2fc42ef65" exitCode=0
Sep 11 20:20:46.853228 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.853065    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"
Sep 11 20:20:46.853228 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.853100    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng" event={"ID":"27d886d5-3619-48ac-9a1c-42f49ee7ba9f","Type":"ContainerDied","Data":"4c02cb48a7db8115ff725f5518fc99234197081535f8fd4410d048d2fc42ef65"}
Sep 11 20:20:46.853228 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.853145    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng" event={"ID":"27d886d5-3619-48ac-9a1c-42f49ee7ba9f","Type":"ContainerDied","Data":"ac49fdb542ede4b71d51bfd34683ae002461e90a0dfc5ab0d3748671ed0c7357"}
Sep 11 20:20:46.853228 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.853163    2580 scope.go:117] "RemoveContainer" containerID="4c02cb48a7db8115ff725f5518fc99234197081535f8fd4410d048d2fc42ef65"
Sep 11 20:20:46.854539 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.854520    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62pvz\" (UniqueName: \"kubernetes.io/projected/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-kube-api-access-62pvz\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:20:46.854634 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.854543    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-dshm\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:20:46.854634 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.854557    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27d886d5-3619-48ac-9a1c-42f49ee7ba9f-tls-certs\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:20:46.865239 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.865223    2580 scope.go:117] "RemoveContainer" containerID="4c02cb48a7db8115ff725f5518fc99234197081535f8fd4410d048d2fc42ef65"
Sep 11 20:20:46.865980 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:20:46.865484    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c02cb48a7db8115ff725f5518fc99234197081535f8fd4410d048d2fc42ef65\": container with ID starting with 4c02cb48a7db8115ff725f5518fc99234197081535f8fd4410d048d2fc42ef65 not found: ID does not exist" containerID="4c02cb48a7db8115ff725f5518fc99234197081535f8fd4410d048d2fc42ef65"
Sep 11 20:20:46.866039 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.865991    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c02cb48a7db8115ff725f5518fc99234197081535f8fd4410d048d2fc42ef65"} err="failed to get container status \"4c02cb48a7db8115ff725f5518fc99234197081535f8fd4410d048d2fc42ef65\": rpc error: code = NotFound desc = could not find container \"4c02cb48a7db8115ff725f5518fc99234197081535f8fd4410d048d2fc42ef65\": container with ID starting with 4c02cb48a7db8115ff725f5518fc99234197081535f8fd4410d048d2fc42ef65 not found: ID does not exist"
Sep 11 20:20:46.878587 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.878562    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"]
Sep 11 20:20:46.880944 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:46.880927    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-279dba/noten-model-279dba-kserve-6d7656f755-qqzng"]
Sep 11 20:20:47.424434 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:47.424397    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f9558-ppkjj"]
Sep 11 20:20:47.424829 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:47.424816    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27d886d5-3619-48ac-9a1c-42f49ee7ba9f" containerName="main"
Sep 11 20:20:47.424829 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:47.424830    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="27d886d5-3619-48ac-9a1c-42f49ee7ba9f" containerName="main"
Sep 11 20:20:47.424901 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:47.424897    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="27d886d5-3619-48ac-9a1c-42f49ee7ba9f" containerName="main"
Sep 11 20:20:47.429211 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:47.429194    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f9558-ppkjj"
Sep 11 20:20:47.431929 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:47.431912    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7dbs5\""
Sep 11 20:20:47.432026 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:47.431981    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:20:47.441085 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:47.441063    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f9558-ppkjj"]
Sep 11 20:20:47.560674 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:47.560638    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d29a8c2-5ef2-4bc0-b66a-b843b021f94c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4f9558-ppkjj\" (UID: \"4d29a8c2-5ef2-4bc0-b66a-b843b021f94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f9558-ppkjj"
Sep 11 20:20:47.661853 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:47.661823    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d29a8c2-5ef2-4bc0-b66a-b843b021f94c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4f9558-ppkjj\" (UID: \"4d29a8c2-5ef2-4bc0-b66a-b843b021f94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f9558-ppkjj"
Sep 11 20:20:47.662415 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:47.662396    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d29a8c2-5ef2-4bc0-b66a-b843b021f94c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4f9558-ppkjj\" (UID: \"4d29a8c2-5ef2-4bc0-b66a-b843b021f94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f9558-ppkjj"
Sep 11 20:20:47.739768 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:47.739731    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f9558-ppkjj"
Sep 11 20:20:48.014979 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:48.014949    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f9558-ppkjj"]
Sep 11 20:20:48.016833 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:20:48.016797    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d29a8c2_5ef2_4bc0_b66a_b843b021f94c.slice/crio-dd54ac15c37fac07811f857b2f2f310e998dde3596ace0cc6507ab17c411820d WatchSource:0}: Error finding container dd54ac15c37fac07811f857b2f2f310e998dde3596ace0cc6507ab17c411820d: Status 404 returned error can't find the container with id dd54ac15c37fac07811f857b2f2f310e998dde3596ace0cc6507ab17c411820d
Sep 11 20:20:48.441368 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:48.441332    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27d886d5-3619-48ac-9a1c-42f49ee7ba9f" path="/var/lib/kubelet/pods/27d886d5-3619-48ac-9a1c-42f49ee7ba9f/volumes"
Sep 11 20:20:48.868996 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:48.868959    2580 generic.go:356] "Generic (PLEG): container finished" podID="4d29a8c2-5ef2-4bc0-b66a-b843b021f94c" containerID="855545f15ab65fc8fa0ed2842278cf50e7c01ea3779e11150d77a073bf52c361" exitCode=0
Sep 11 20:20:48.869171 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:48.869003    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f9558-ppkjj" event={"ID":"4d29a8c2-5ef2-4bc0-b66a-b843b021f94c","Type":"ContainerDied","Data":"855545f15ab65fc8fa0ed2842278cf50e7c01ea3779e11150d77a073bf52c361"}
Sep 11 20:20:48.869171 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:48.869058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f9558-ppkjj" event={"ID":"4d29a8c2-5ef2-4bc0-b66a-b843b021f94c","Type":"ContainerStarted","Data":"dd54ac15c37fac07811f857b2f2f310e998dde3596ace0cc6507ab17c411820d"}
Sep 11 20:20:49.025311 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:49.025212    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:20:49.025311 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:49.025280    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:20:49.037897 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:49.037874    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:20:49.890194 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:49.890163    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:20:50.017014 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:50.016990    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f9558-ppkjj"
Sep 11 20:20:50.186923 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:50.186899    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d29a8c2-5ef2-4bc0-b66a-b843b021f94c-maas-api-service-ca\") pod \"4d29a8c2-5ef2-4bc0-b66a-b843b021f94c\" (UID: \"4d29a8c2-5ef2-4bc0-b66a-b843b021f94c\") "
Sep 11 20:20:50.187252 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:50.187228    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d29a8c2-5ef2-4bc0-b66a-b843b021f94c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4d29a8c2-5ef2-4bc0-b66a-b843b021f94c" (UID: "4d29a8c2-5ef2-4bc0-b66a-b843b021f94c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:20:50.288424 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:50.288381    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d29a8c2-5ef2-4bc0-b66a-b843b021f94c-maas-api-service-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:20:50.882963 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:50.882824    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f9558-ppkjj"
Sep 11 20:20:50.882963 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:50.882861    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f9558-ppkjj" event={"ID":"4d29a8c2-5ef2-4bc0-b66a-b843b021f94c","Type":"ContainerDied","Data":"dd54ac15c37fac07811f857b2f2f310e998dde3596ace0cc6507ab17c411820d"}
Sep 11 20:20:50.882963 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:50.882920    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd54ac15c37fac07811f857b2f2f310e998dde3596ace0cc6507ab17c411820d"
Sep 11 20:20:57.977886 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:57.977851    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f9558-29819295-vzdww"]
Sep 11 20:20:57.982442 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:57.982417    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f9558-29819295-vzdww"]
Sep 11 20:20:58.006093 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:58.006060    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl"]
Sep 11 20:20:58.006377 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:58.006350    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl" podUID="6e079451-d9ce-46f4-ae94-012e2b81242c" containerName="payload-processing" containerID="cri-o://f2aba01ab42a8ced212cd0f834a1d4a4519921823b757c017df62fb9020775d0" gracePeriod=30
Sep 11 20:20:58.180350 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:58.180318    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f9558-ppkjj"]
Sep 11 20:20:58.182709 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:58.182681    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4f9558-ppkjj"]
Sep 11 20:20:58.270027 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:58.270000    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl"
Sep 11 20:20:58.361168 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:58.361129    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cp7c9\" (UniqueName: \"kubernetes.io/projected/6e079451-d9ce-46f4-ae94-012e2b81242c-kube-api-access-cp7c9\") pod \"6e079451-d9ce-46f4-ae94-012e2b81242c\" (UID: \"6e079451-d9ce-46f4-ae94-012e2b81242c\") "
Sep 11 20:20:58.361364 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:58.361226    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e079451-d9ce-46f4-ae94-012e2b81242c-plugins-config-volume\") pod \"6e079451-d9ce-46f4-ae94-012e2b81242c\" (UID: \"6e079451-d9ce-46f4-ae94-012e2b81242c\") "
Sep 11 20:20:58.361656 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:58.361607    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e079451-d9ce-46f4-ae94-012e2b81242c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e079451-d9ce-46f4-ae94-012e2b81242c" (UID: "6e079451-d9ce-46f4-ae94-012e2b81242c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:20:58.363309 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:58.363290    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e079451-d9ce-46f4-ae94-012e2b81242c-kube-api-access-cp7c9" (OuterVolumeSpecName: "kube-api-access-cp7c9") pod "6e079451-d9ce-46f4-ae94-012e2b81242c" (UID: "6e079451-d9ce-46f4-ae94-012e2b81242c"). InnerVolumeSpecName "kube-api-access-cp7c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:20:58.442193 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:58.442157    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d29a8c2-5ef2-4bc0-b66a-b843b021f94c" path="/var/lib/kubelet/pods/4d29a8c2-5ef2-4bc0-b66a-b843b021f94c/volumes"
Sep 11 20:20:58.442449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:58.442437    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecc0a542-ae11-4b7f-ad11-8431ac73871c" path="/var/lib/kubelet/pods/ecc0a542-ae11-4b7f-ad11-8431ac73871c/volumes"
Sep 11 20:20:58.462063 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:58.462037    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e079451-d9ce-46f4-ae94-012e2b81242c-plugins-config-volume\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:20:58.462153 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:58.462060    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp7c9\" (UniqueName: \"kubernetes.io/projected/6e079451-d9ce-46f4-ae94-012e2b81242c-kube-api-access-cp7c9\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:20:58.943199 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:58.943168    2580 generic.go:356] "Generic (PLEG): container finished" podID="6e079451-d9ce-46f4-ae94-012e2b81242c" containerID="f2aba01ab42a8ced212cd0f834a1d4a4519921823b757c017df62fb9020775d0" exitCode=0
Sep 11 20:20:58.943385 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:58.943217    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl"
Sep 11 20:20:58.943385 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:58.943257    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl" event={"ID":"6e079451-d9ce-46f4-ae94-012e2b81242c","Type":"ContainerDied","Data":"f2aba01ab42a8ced212cd0f834a1d4a4519921823b757c017df62fb9020775d0"}
Sep 11 20:20:58.943385 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:58.943316    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl" event={"ID":"6e079451-d9ce-46f4-ae94-012e2b81242c","Type":"ContainerDied","Data":"f0654209f969eff4b4c15e16c2580955de5132d7e22ee9b9c0e0ea8dd9f87ef5"}
Sep 11 20:20:58.943385 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:58.943346    2580 scope.go:117] "RemoveContainer" containerID="f2aba01ab42a8ced212cd0f834a1d4a4519921823b757c017df62fb9020775d0"
Sep 11 20:20:58.955344 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:58.955310    2580 scope.go:117] "RemoveContainer" containerID="f2aba01ab42a8ced212cd0f834a1d4a4519921823b757c017df62fb9020775d0"
Sep 11 20:20:58.955648 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:20:58.955628    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2aba01ab42a8ced212cd0f834a1d4a4519921823b757c017df62fb9020775d0\": container with ID starting with f2aba01ab42a8ced212cd0f834a1d4a4519921823b757c017df62fb9020775d0 not found: ID does not exist" containerID="f2aba01ab42a8ced212cd0f834a1d4a4519921823b757c017df62fb9020775d0"
Sep 11 20:20:58.955708 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:58.955657    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2aba01ab42a8ced212cd0f834a1d4a4519921823b757c017df62fb9020775d0"} err="failed to get container status \"f2aba01ab42a8ced212cd0f834a1d4a4519921823b757c017df62fb9020775d0\": rpc error: code = NotFound desc = could not find container \"f2aba01ab42a8ced212cd0f834a1d4a4519921823b757c017df62fb9020775d0\": container with ID starting with f2aba01ab42a8ced212cd0f834a1d4a4519921823b757c017df62fb9020775d0 not found: ID does not exist"
Sep 11 20:20:58.968811 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:58.968777    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl"]
Sep 11 20:20:58.968944 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:58.968831    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4f9558-55db69f777-fhnwl"]
Sep 11 20:20:59.992661 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:20:59.992602    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64" podUID="75d16ee9-8f38-424b-bcd8-224756e9607f" containerName="main" containerID="cri-o://34c1f73ed9556e9e8e717494f85b309d3a876fa32f36563218d73fe880a71697" gracePeriod=45
Sep 11 20:21:00.252304 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.252241    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:21:00.278924 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.278894    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75d16ee9-8f38-424b-bcd8-224756e9607f-tmp-dir\") pod \"75d16ee9-8f38-424b-bcd8-224756e9607f\" (UID: \"75d16ee9-8f38-424b-bcd8-224756e9607f\") "
Sep 11 20:21:00.279046 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.278996    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zj2b5\" (UniqueName: \"kubernetes.io/projected/75d16ee9-8f38-424b-bcd8-224756e9607f-kube-api-access-zj2b5\") pod \"75d16ee9-8f38-424b-bcd8-224756e9607f\" (UID: \"75d16ee9-8f38-424b-bcd8-224756e9607f\") "
Sep 11 20:21:00.279046 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.279030    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75d16ee9-8f38-424b-bcd8-224756e9607f-dshm\") pod \"75d16ee9-8f38-424b-bcd8-224756e9607f\" (UID: \"75d16ee9-8f38-424b-bcd8-224756e9607f\") "
Sep 11 20:21:00.279124 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.279085    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75d16ee9-8f38-424b-bcd8-224756e9607f-home\") pod \"75d16ee9-8f38-424b-bcd8-224756e9607f\" (UID: \"75d16ee9-8f38-424b-bcd8-224756e9607f\") "
Sep 11 20:21:00.279159 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.279137    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75d16ee9-8f38-424b-bcd8-224756e9607f-tls-certs\") pod \"75d16ee9-8f38-424b-bcd8-224756e9607f\" (UID: \"75d16ee9-8f38-424b-bcd8-224756e9607f\") "
Sep 11 20:21:00.279196 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.279162    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75d16ee9-8f38-424b-bcd8-224756e9607f-model-cache\") pod \"75d16ee9-8f38-424b-bcd8-224756e9607f\" (UID: \"75d16ee9-8f38-424b-bcd8-224756e9607f\") "
Sep 11 20:21:00.279687 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.279647    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d16ee9-8f38-424b-bcd8-224756e9607f-home" (OuterVolumeSpecName: "home") pod "75d16ee9-8f38-424b-bcd8-224756e9607f" (UID: "75d16ee9-8f38-424b-bcd8-224756e9607f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:21:00.279854 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.279703    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d16ee9-8f38-424b-bcd8-224756e9607f-model-cache" (OuterVolumeSpecName: "model-cache") pod "75d16ee9-8f38-424b-bcd8-224756e9607f" (UID: "75d16ee9-8f38-424b-bcd8-224756e9607f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:21:00.279983 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.279662    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d16ee9-8f38-424b-bcd8-224756e9607f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "75d16ee9-8f38-424b-bcd8-224756e9607f" (UID: "75d16ee9-8f38-424b-bcd8-224756e9607f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:21:00.281714 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.281684    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75d16ee9-8f38-424b-bcd8-224756e9607f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75d16ee9-8f38-424b-bcd8-224756e9607f" (UID: "75d16ee9-8f38-424b-bcd8-224756e9607f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:21:00.281834 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.281811    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d16ee9-8f38-424b-bcd8-224756e9607f-kube-api-access-zj2b5" (OuterVolumeSpecName: "kube-api-access-zj2b5") pod "75d16ee9-8f38-424b-bcd8-224756e9607f" (UID: "75d16ee9-8f38-424b-bcd8-224756e9607f"). InnerVolumeSpecName "kube-api-access-zj2b5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:21:00.281882 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.281863    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d16ee9-8f38-424b-bcd8-224756e9607f-dshm" (OuterVolumeSpecName: "dshm") pod "75d16ee9-8f38-424b-bcd8-224756e9607f" (UID: "75d16ee9-8f38-424b-bcd8-224756e9607f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:21:00.376742 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.376709    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"]
Sep 11 20:21:00.377131 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.377119    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75d16ee9-8f38-424b-bcd8-224756e9607f" containerName="main"
Sep 11 20:21:00.377173 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.377133    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="75d16ee9-8f38-424b-bcd8-224756e9607f" containerName="main"
Sep 11 20:21:00.377173 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.377142    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d29a8c2-5ef2-4bc0-b66a-b843b021f94c" containerName="revoke-keys"
Sep 11 20:21:00.377173 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.377148    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d29a8c2-5ef2-4bc0-b66a-b843b021f94c" containerName="revoke-keys"
Sep 11 20:21:00.377173 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.377160    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e079451-d9ce-46f4-ae94-012e2b81242c" containerName="payload-processing"
Sep 11 20:21:00.377173 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.377166    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e079451-d9ce-46f4-ae94-012e2b81242c" containerName="payload-processing"
Sep 11 20:21:00.377364 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.377243    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e079451-d9ce-46f4-ae94-012e2b81242c" containerName="payload-processing"
Sep 11 20:21:00.377364 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.377253    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="75d16ee9-8f38-424b-bcd8-224756e9607f" containerName="main"
Sep 11 20:21:00.377364 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.377263    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d29a8c2-5ef2-4bc0-b66a-b843b021f94c" containerName="revoke-keys"
Sep 11 20:21:00.380243 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.380217    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj2b5\" (UniqueName: \"kubernetes.io/projected/75d16ee9-8f38-424b-bcd8-224756e9607f-kube-api-access-zj2b5\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:00.380243 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.380245    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75d16ee9-8f38-424b-bcd8-224756e9607f-dshm\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:00.380401 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.380264    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75d16ee9-8f38-424b-bcd8-224756e9607f-home\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:00.380401 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.380273    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75d16ee9-8f38-424b-bcd8-224756e9607f-tls-certs\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:00.380401 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.380283    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75d16ee9-8f38-424b-bcd8-224756e9607f-model-cache\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:00.380401 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.380290    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75d16ee9-8f38-424b-bcd8-224756e9607f-tmp-dir\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:00.380553 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.380523    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:21:00.383240 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.383214    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d11bc81f\"/\"kube-root-ca.crt\""
Sep 11 20:21:00.383360 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.383320    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d11bc81f\"/\"openshift-service-ca.crt\""
Sep 11 20:21:00.383425 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.383367    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d11bc81f\"/\"e2e-lifecycle-model-d11bc81f-kserve-self-signed-certs\""
Sep 11 20:21:00.383632 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.383615    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d11bc81f\"/\"default-dockercfg-88wn9\""
Sep 11 20:21:00.392382 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.392355    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"]
Sep 11 20:21:00.442771 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.442743    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e079451-d9ce-46f4-ae94-012e2b81242c" path="/var/lib/kubelet/pods/6e079451-d9ce-46f4-ae94-012e2b81242c/volumes"
Sep 11 20:21:00.481208 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.481176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73739b33-c83d-41c0-8fc7-be15e0b122e2-home\") pod \"e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw\" (UID: \"73739b33-c83d-41c0-8fc7-be15e0b122e2\") " pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:21:00.481359 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.481244    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73739b33-c83d-41c0-8fc7-be15e0b122e2-model-cache\") pod \"e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw\" (UID: \"73739b33-c83d-41c0-8fc7-be15e0b122e2\") " pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:21:00.481359 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.481277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73739b33-c83d-41c0-8fc7-be15e0b122e2-tls-certs\") pod \"e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw\" (UID: \"73739b33-c83d-41c0-8fc7-be15e0b122e2\") " pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:21:00.481359 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.481319    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jxt6\" (UniqueName: \"kubernetes.io/projected/73739b33-c83d-41c0-8fc7-be15e0b122e2-kube-api-access-7jxt6\") pod \"e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw\" (UID: \"73739b33-c83d-41c0-8fc7-be15e0b122e2\") " pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:21:00.481359 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.481346    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73739b33-c83d-41c0-8fc7-be15e0b122e2-tmp-dir\") pod \"e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw\" (UID: \"73739b33-c83d-41c0-8fc7-be15e0b122e2\") " pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:21:00.481569 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.481385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73739b33-c83d-41c0-8fc7-be15e0b122e2-dshm\") pod \"e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw\" (UID: \"73739b33-c83d-41c0-8fc7-be15e0b122e2\") " pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:21:00.581982 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.581878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73739b33-c83d-41c0-8fc7-be15e0b122e2-home\") pod \"e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw\" (UID: \"73739b33-c83d-41c0-8fc7-be15e0b122e2\") " pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:21:00.581982 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.581963    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73739b33-c83d-41c0-8fc7-be15e0b122e2-model-cache\") pod \"e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw\" (UID: \"73739b33-c83d-41c0-8fc7-be15e0b122e2\") " pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:21:00.582249 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.581998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73739b33-c83d-41c0-8fc7-be15e0b122e2-tls-certs\") pod \"e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw\" (UID: \"73739b33-c83d-41c0-8fc7-be15e0b122e2\") " pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:21:00.582249 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.582135    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jxt6\" (UniqueName: \"kubernetes.io/projected/73739b33-c83d-41c0-8fc7-be15e0b122e2-kube-api-access-7jxt6\") pod \"e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw\" (UID: \"73739b33-c83d-41c0-8fc7-be15e0b122e2\") " pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:21:00.582249 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.582175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73739b33-c83d-41c0-8fc7-be15e0b122e2-tmp-dir\") pod \"e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw\" (UID: \"73739b33-c83d-41c0-8fc7-be15e0b122e2\") " pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:21:00.582249 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.582223    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73739b33-c83d-41c0-8fc7-be15e0b122e2-dshm\") pod \"e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw\" (UID: \"73739b33-c83d-41c0-8fc7-be15e0b122e2\") " pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:21:00.582454 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.582252    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73739b33-c83d-41c0-8fc7-be15e0b122e2-home\") pod \"e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw\" (UID: \"73739b33-c83d-41c0-8fc7-be15e0b122e2\") " pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:21:00.582454 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.582310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73739b33-c83d-41c0-8fc7-be15e0b122e2-model-cache\") pod \"e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw\" (UID: \"73739b33-c83d-41c0-8fc7-be15e0b122e2\") " pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:21:00.582454 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.582431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73739b33-c83d-41c0-8fc7-be15e0b122e2-tmp-dir\") pod \"e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw\" (UID: \"73739b33-c83d-41c0-8fc7-be15e0b122e2\") " pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:21:00.584315 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.584292    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73739b33-c83d-41c0-8fc7-be15e0b122e2-dshm\") pod \"e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw\" (UID: \"73739b33-c83d-41c0-8fc7-be15e0b122e2\") " pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:21:00.584546 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.584515    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73739b33-c83d-41c0-8fc7-be15e0b122e2-tls-certs\") pod \"e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw\" (UID: \"73739b33-c83d-41c0-8fc7-be15e0b122e2\") " pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:21:00.590422 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.590401    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jxt6\" (UniqueName: \"kubernetes.io/projected/73739b33-c83d-41c0-8fc7-be15e0b122e2-kube-api-access-7jxt6\") pod \"e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw\" (UID: \"73739b33-c83d-41c0-8fc7-be15e0b122e2\") " pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:21:00.690178 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.690147    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:21:00.958632 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.958604    2580 generic.go:356] "Generic (PLEG): container finished" podID="75d16ee9-8f38-424b-bcd8-224756e9607f" containerID="34c1f73ed9556e9e8e717494f85b309d3a876fa32f36563218d73fe880a71697" exitCode=0
Sep 11 20:21:00.958791 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.958644    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"
Sep 11 20:21:00.958791 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.958684    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64" event={"ID":"75d16ee9-8f38-424b-bcd8-224756e9607f","Type":"ContainerDied","Data":"34c1f73ed9556e9e8e717494f85b309d3a876fa32f36563218d73fe880a71697"}
Sep 11 20:21:00.958791 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.958728    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64" event={"ID":"75d16ee9-8f38-424b-bcd8-224756e9607f","Type":"ContainerDied","Data":"a9b09cb134dbae8868936b338adf64d0d424a7c3f774f63cb2870d0f0010c7ad"}
Sep 11 20:21:00.958791 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.958742    2580 scope.go:117] "RemoveContainer" containerID="34c1f73ed9556e9e8e717494f85b309d3a876fa32f36563218d73fe880a71697"
Sep 11 20:21:00.968729 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.968705    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"]
Sep 11 20:21:00.969666 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.969644    2580 scope.go:117] "RemoveContainer" containerID="34c1f73ed9556e9e8e717494f85b309d3a876fa32f36563218d73fe880a71697"
Sep 11 20:21:00.969959 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:21:00.969938    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34c1f73ed9556e9e8e717494f85b309d3a876fa32f36563218d73fe880a71697\": container with ID starting with 34c1f73ed9556e9e8e717494f85b309d3a876fa32f36563218d73fe880a71697 not found: ID does not exist" containerID="34c1f73ed9556e9e8e717494f85b309d3a876fa32f36563218d73fe880a71697"
Sep 11 20:21:00.970021 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.969970    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34c1f73ed9556e9e8e717494f85b309d3a876fa32f36563218d73fe880a71697"} err="failed to get container status \"34c1f73ed9556e9e8e717494f85b309d3a876fa32f36563218d73fe880a71697\": rpc error: code = NotFound desc = could not find container \"34c1f73ed9556e9e8e717494f85b309d3a876fa32f36563218d73fe880a71697\": container with ID starting with 34c1f73ed9556e9e8e717494f85b309d3a876fa32f36563218d73fe880a71697 not found: ID does not exist"
Sep 11 20:21:00.971219 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:21:00.971196    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73739b33_c83d_41c0_8fc7_be15e0b122e2.slice/crio-481d0f29604a3ac80c10fa261ca5c4539256e049d6d1b4a58f873b374bf2455f WatchSource:0}: Error finding container 481d0f29604a3ac80c10fa261ca5c4539256e049d6d1b4a58f873b374bf2455f: Status 404 returned error can't find the container with id 481d0f29604a3ac80c10fa261ca5c4539256e049d6d1b4a58f873b374bf2455f
Sep 11 20:21:00.981020 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.980998    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"]
Sep 11 20:21:00.989317 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:00.989293    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4f9558/test-model-4f9558-kserve-6696b864c-lxn64"]
Sep 11 20:21:01.967485 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:01.967444    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw" event={"ID":"73739b33-c83d-41c0-8fc7-be15e0b122e2","Type":"ContainerStarted","Data":"ab14517d6a8b8525624e13c717be348323766cb30b65af7f4d170137e8f814d8"}
Sep 11 20:21:01.967883 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:01.967518    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw" event={"ID":"73739b33-c83d-41c0-8fc7-be15e0b122e2","Type":"ContainerStarted","Data":"481d0f29604a3ac80c10fa261ca5c4539256e049d6d1b4a58f873b374bf2455f"}
Sep 11 20:21:01.985846 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:01.985800    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw" podStartSLOduration=1.985785624 podStartE2EDuration="1.985785624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:21:01.983542495 +0000 UTC m=+1231.975704774" watchObservedRunningTime="2026-09-11 20:21:01.985785624 +0000 UTC m=+1231.977947939"
Sep 11 20:21:02.441367 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:02.441333    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75d16ee9-8f38-424b-bcd8-224756e9607f" path="/var/lib/kubelet/pods/75d16ee9-8f38-424b-bcd8-224756e9607f/volumes"
Sep 11 20:21:03.257560 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:03.257513    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"]
Sep 11 20:21:10.690886 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:10.690850    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:21:10.691299 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:10.690919    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:21:10.703734 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:10.703706    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:21:11.036995 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:11.036918    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:21:11.509777 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:11.509739    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:21:18.278858 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:18.278816    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5" podUID="cf817703-5261-45cf-a8ee-922ca810468e" containerName="main" containerID="cri-o://c1c8f953f3c7e40973cb78d715c94ca4b7ab241a9ef2120275d88ff19bdd6d96" gracePeriod=45
Sep 11 20:21:18.536202 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:18.536133    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:21:18.644806 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:18.644776    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84b4r\" (UniqueName: \"kubernetes.io/projected/cf817703-5261-45cf-a8ee-922ca810468e-kube-api-access-84b4r\") pod \"cf817703-5261-45cf-a8ee-922ca810468e\" (UID: \"cf817703-5261-45cf-a8ee-922ca810468e\") "
Sep 11 20:21:18.644981 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:18.644823    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf817703-5261-45cf-a8ee-922ca810468e-tmp-dir\") pod \"cf817703-5261-45cf-a8ee-922ca810468e\" (UID: \"cf817703-5261-45cf-a8ee-922ca810468e\") "
Sep 11 20:21:18.644981 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:18.644854    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf817703-5261-45cf-a8ee-922ca810468e-model-cache\") pod \"cf817703-5261-45cf-a8ee-922ca810468e\" (UID: \"cf817703-5261-45cf-a8ee-922ca810468e\") "
Sep 11 20:21:18.644981 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:18.644897    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf817703-5261-45cf-a8ee-922ca810468e-tls-certs\") pod \"cf817703-5261-45cf-a8ee-922ca810468e\" (UID: \"cf817703-5261-45cf-a8ee-922ca810468e\") "
Sep 11 20:21:18.644981 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:18.644948    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf817703-5261-45cf-a8ee-922ca810468e-dshm\") pod \"cf817703-5261-45cf-a8ee-922ca810468e\" (UID: \"cf817703-5261-45cf-a8ee-922ca810468e\") "
Sep 11 20:21:18.645193 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:18.645010    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf817703-5261-45cf-a8ee-922ca810468e-home\") pod \"cf817703-5261-45cf-a8ee-922ca810468e\" (UID: \"cf817703-5261-45cf-a8ee-922ca810468e\") "
Sep 11 20:21:18.645193 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:18.645159    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf817703-5261-45cf-a8ee-922ca810468e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cf817703-5261-45cf-a8ee-922ca810468e" (UID: "cf817703-5261-45cf-a8ee-922ca810468e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:21:18.645193 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:18.645174    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf817703-5261-45cf-a8ee-922ca810468e-model-cache" (OuterVolumeSpecName: "model-cache") pod "cf817703-5261-45cf-a8ee-922ca810468e" (UID: "cf817703-5261-45cf-a8ee-922ca810468e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:21:18.645359 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:18.645292    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf817703-5261-45cf-a8ee-922ca810468e-home" (OuterVolumeSpecName: "home") pod "cf817703-5261-45cf-a8ee-922ca810468e" (UID: "cf817703-5261-45cf-a8ee-922ca810468e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:21:18.645404 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:18.645360    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf817703-5261-45cf-a8ee-922ca810468e-tmp-dir\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:18.645404 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:18.645379    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf817703-5261-45cf-a8ee-922ca810468e-model-cache\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:18.647086 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:18.647058    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf817703-5261-45cf-a8ee-922ca810468e-dshm" (OuterVolumeSpecName: "dshm") pod "cf817703-5261-45cf-a8ee-922ca810468e" (UID: "cf817703-5261-45cf-a8ee-922ca810468e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:21:18.647205 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:18.647115    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf817703-5261-45cf-a8ee-922ca810468e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cf817703-5261-45cf-a8ee-922ca810468e" (UID: "cf817703-5261-45cf-a8ee-922ca810468e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:21:18.647205 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:18.647153    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf817703-5261-45cf-a8ee-922ca810468e-kube-api-access-84b4r" (OuterVolumeSpecName: "kube-api-access-84b4r") pod "cf817703-5261-45cf-a8ee-922ca810468e" (UID: "cf817703-5261-45cf-a8ee-922ca810468e"). InnerVolumeSpecName "kube-api-access-84b4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:21:18.746808 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:18.746773    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf817703-5261-45cf-a8ee-922ca810468e-tls-certs\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:18.746808 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:18.746802    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf817703-5261-45cf-a8ee-922ca810468e-dshm\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:18.746808 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:18.746812    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf817703-5261-45cf-a8ee-922ca810468e-home\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:18.747039 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:18.746821    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84b4r\" (UniqueName: \"kubernetes.io/projected/cf817703-5261-45cf-a8ee-922ca810468e-kube-api-access-84b4r\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:19.078398 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:19.078365    2580 generic.go:356] "Generic (PLEG): container finished" podID="cf817703-5261-45cf-a8ee-922ca810468e" containerID="c1c8f953f3c7e40973cb78d715c94ca4b7ab241a9ef2120275d88ff19bdd6d96" exitCode=0
Sep 11 20:21:19.078599 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:19.078407    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"
Sep 11 20:21:19.078599 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:19.078449    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5" event={"ID":"cf817703-5261-45cf-a8ee-922ca810468e","Type":"ContainerDied","Data":"c1c8f953f3c7e40973cb78d715c94ca4b7ab241a9ef2120275d88ff19bdd6d96"}
Sep 11 20:21:19.078599 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:19.078512    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5" event={"ID":"cf817703-5261-45cf-a8ee-922ca810468e","Type":"ContainerDied","Data":"b0364912347a2240ba830b19b193b124520f21956d64f3e6dd751c6e61523114"}
Sep 11 20:21:19.078599 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:19.078534    2580 scope.go:117] "RemoveContainer" containerID="c1c8f953f3c7e40973cb78d715c94ca4b7ab241a9ef2120275d88ff19bdd6d96"
Sep 11 20:21:19.088154 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:19.088135    2580 scope.go:117] "RemoveContainer" containerID="c1c8f953f3c7e40973cb78d715c94ca4b7ab241a9ef2120275d88ff19bdd6d96"
Sep 11 20:21:19.088437 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:21:19.088420    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c8f953f3c7e40973cb78d715c94ca4b7ab241a9ef2120275d88ff19bdd6d96\": container with ID starting with c1c8f953f3c7e40973cb78d715c94ca4b7ab241a9ef2120275d88ff19bdd6d96 not found: ID does not exist" containerID="c1c8f953f3c7e40973cb78d715c94ca4b7ab241a9ef2120275d88ff19bdd6d96"
Sep 11 20:21:19.088487 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:19.088444    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c8f953f3c7e40973cb78d715c94ca4b7ab241a9ef2120275d88ff19bdd6d96"} err="failed to get container status \"c1c8f953f3c7e40973cb78d715c94ca4b7ab241a9ef2120275d88ff19bdd6d96\": rpc error: code = NotFound desc = could not find container \"c1c8f953f3c7e40973cb78d715c94ca4b7ab241a9ef2120275d88ff19bdd6d96\": container with ID starting with c1c8f953f3c7e40973cb78d715c94ca4b7ab241a9ef2120275d88ff19bdd6d96 not found: ID does not exist"
Sep 11 20:21:19.099566 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:19.099545    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"]
Sep 11 20:21:19.103487 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:19.103466    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-22fc84/e2e-conflict-b-cb87ecd4-kserve-68b48754c9-x6rg5"]
Sep 11 20:21:20.441683 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:20.441650    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf817703-5261-45cf-a8ee-922ca810468e" path="/var/lib/kubelet/pods/cf817703-5261-45cf-a8ee-922ca810468e/volumes"
Sep 11 20:21:29.130831 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:29.130795    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9d9abb-x2nd7"]
Sep 11 20:21:29.131231 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:29.131194    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf817703-5261-45cf-a8ee-922ca810468e" containerName="main"
Sep 11 20:21:29.131231 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:29.131207    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf817703-5261-45cf-a8ee-922ca810468e" containerName="main"
Sep 11 20:21:29.131308 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:29.131274    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf817703-5261-45cf-a8ee-922ca810468e" containerName="main"
Sep 11 20:21:29.135744 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:29.135727    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9d9abb-x2nd7"
Sep 11 20:21:29.138812 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:29.138617    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7dbs5\""
Sep 11 20:21:29.139024 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:29.139005    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:21:29.141017 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:29.140994    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9d9abb-x2nd7"]
Sep 11 20:21:29.243168 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:29.243126    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1030e004-bc97-46ac-8e16-1ae0b0b3021c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9d9abb-x2nd7\" (UID: \"1030e004-bc97-46ac-8e16-1ae0b0b3021c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9d9abb-x2nd7"
Sep 11 20:21:29.344129 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:29.344096    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1030e004-bc97-46ac-8e16-1ae0b0b3021c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9d9abb-x2nd7\" (UID: \"1030e004-bc97-46ac-8e16-1ae0b0b3021c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9d9abb-x2nd7"
Sep 11 20:21:29.344721 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:29.344703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1030e004-bc97-46ac-8e16-1ae0b0b3021c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9d9abb-x2nd7\" (UID: \"1030e004-bc97-46ac-8e16-1ae0b0b3021c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9d9abb-x2nd7"
Sep 11 20:21:29.450167 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:29.450136    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9d9abb-x2nd7"
Sep 11 20:21:29.722548 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:29.722458    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9d9abb-x2nd7"]
Sep 11 20:21:29.725792 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:21:29.725765    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1030e004_bc97_46ac_8e16_1ae0b0b3021c.slice/crio-ce370d788d13202847d0faed38930ad4eb69c712a96421ab1519f1c20040bf61 WatchSource:0}: Error finding container ce370d788d13202847d0faed38930ad4eb69c712a96421ab1519f1c20040bf61: Status 404 returned error can't find the container with id ce370d788d13202847d0faed38930ad4eb69c712a96421ab1519f1c20040bf61
Sep 11 20:21:30.152765 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:30.152725    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9d9abb-x2nd7" event={"ID":"1030e004-bc97-46ac-8e16-1ae0b0b3021c","Type":"ContainerStarted","Data":"f7732a66540f9f03a8d012d0de010c90ea012cc2841f60b481d4333cd726125c"}
Sep 11 20:21:30.153170 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:30.152779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9d9abb-x2nd7" event={"ID":"1030e004-bc97-46ac-8e16-1ae0b0b3021c","Type":"ContainerStarted","Data":"ce370d788d13202847d0faed38930ad4eb69c712a96421ab1519f1c20040bf61"}
Sep 11 20:21:30.175034 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:30.174986    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9d9abb-x2nd7" podStartSLOduration=1.174969979 podStartE2EDuration="1.174969979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:21:30.173084039 +0000 UTC m=+1260.165246334" watchObservedRunningTime="2026-09-11 20:21:30.174969979 +0000 UTC m=+1260.167132249"
Sep 11 20:21:30.593622 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:30.593591    2580 scope.go:117] "RemoveContainer" containerID="134e0d3718724dac1021549774c3828c6c213e1b18db59afd0abd883b9ac974b"
Sep 11 20:21:30.603033 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:30.603013    2580 scope.go:117] "RemoveContainer" containerID="f131875f48e0fb3924a32b460905a90c501acb49cd8773a99c1d9af93a57703d"
Sep 11 20:21:30.611107 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:30.611090    2580 scope.go:117] "RemoveContainer" containerID="cfc3845478758cd4a7adf263d302cc8f2a9d65a72a7a1cb4e276ddf62f5b34ee"
Sep 11 20:21:30.618830 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:30.618812    2580 scope.go:117] "RemoveContainer" containerID="2568423964dae6f8bd6ae0ff15b7068359bd74e97cb2b4ff2d88b462fe30a256"
Sep 11 20:21:30.626432 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:30.626411    2580 scope.go:117] "RemoveContainer" containerID="61155eba929bf03cca696ab03ad560bc3db6a46ff39373d9ce85e06622a008e2"
Sep 11 20:21:30.633981 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:30.633963    2580 scope.go:117] "RemoveContainer" containerID="19926e0307a82c7f5109de36b9c294d8e16c7c8ebd6ac9a6c529ac3ef808e705"
Sep 11 20:21:31.160330 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:31.160296    2580 generic.go:356] "Generic (PLEG): container finished" podID="1030e004-bc97-46ac-8e16-1ae0b0b3021c" containerID="f7732a66540f9f03a8d012d0de010c90ea012cc2841f60b481d4333cd726125c" exitCode=0
Sep 11 20:21:31.160739 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:31.160349    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9d9abb-x2nd7" event={"ID":"1030e004-bc97-46ac-8e16-1ae0b0b3021c","Type":"ContainerDied","Data":"f7732a66540f9f03a8d012d0de010c90ea012cc2841f60b481d4333cd726125c"}
Sep 11 20:21:32.295737 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:32.295710    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9d9abb-x2nd7"
Sep 11 20:21:32.373831 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:32.373798    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1030e004-bc97-46ac-8e16-1ae0b0b3021c-maas-api-service-ca\") pod \"1030e004-bc97-46ac-8e16-1ae0b0b3021c\" (UID: \"1030e004-bc97-46ac-8e16-1ae0b0b3021c\") "
Sep 11 20:21:32.374186 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:32.374154    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1030e004-bc97-46ac-8e16-1ae0b0b3021c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1030e004-bc97-46ac-8e16-1ae0b0b3021c" (UID: "1030e004-bc97-46ac-8e16-1ae0b0b3021c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:21:32.474474 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:32.474445    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1030e004-bc97-46ac-8e16-1ae0b0b3021c-maas-api-service-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:33.176284 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:33.176253    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9d9abb-x2nd7"
Sep 11 20:21:33.176458 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:33.176292    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9d9abb-x2nd7" event={"ID":"1030e004-bc97-46ac-8e16-1ae0b0b3021c","Type":"ContainerDied","Data":"ce370d788d13202847d0faed38930ad4eb69c712a96421ab1519f1c20040bf61"}
Sep 11 20:21:33.176458 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:33.176342    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce370d788d13202847d0faed38930ad4eb69c712a96421ab1519f1c20040bf61"
Sep 11 20:21:34.020821 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:34.020785    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"]
Sep 11 20:21:34.041261 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:34.041233    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"]
Sep 11 20:21:37.513853 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:37.513810    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:21:40.892375 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:40.892341    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb"]
Sep 11 20:21:40.892782 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:40.892745    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1030e004-bc97-46ac-8e16-1ae0b0b3021c" containerName="revoke-keys"
Sep 11 20:21:40.892782 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:40.892759    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1030e004-bc97-46ac-8e16-1ae0b0b3021c" containerName="revoke-keys"
Sep 11 20:21:40.892855 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:40.892838    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1030e004-bc97-46ac-8e16-1ae0b0b3021c" containerName="revoke-keys"
Sep 11 20:21:40.896320 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:40.896302    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb"
Sep 11 20:21:40.898844 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:40.898813    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-092e1a\""
Sep 11 20:21:40.898998 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:40.898980    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-092e1a-dockercfg-t7d9h\""
Sep 11 20:21:40.904432 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:40.904409    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb"]
Sep 11 20:21:40.918920 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:40.918893    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm"]
Sep 11 20:21:40.922420 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:40.922392    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm"
Sep 11 20:21:40.934503 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:40.934458    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm"]
Sep 11 20:21:41.053256 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:41.053222    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8zxw\" (UniqueName: \"kubernetes.io/projected/cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e-kube-api-access-f8zxw\") pod \"payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb\" (UID: \"cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb"
Sep 11 20:21:41.053461 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:41.053293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv9f5\" (UniqueName: \"kubernetes.io/projected/dcda5f1e-d036-45a2-bf00-60539701c98e-kube-api-access-gv9f5\") pod \"payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm\" (UID: \"dcda5f1e-d036-45a2-bf00-60539701c98e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm"
Sep 11 20:21:41.053461 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:41.053326    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb\" (UID: \"cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb"
Sep 11 20:21:41.053461 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:41.053359    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcda5f1e-d036-45a2-bf00-60539701c98e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm\" (UID: \"dcda5f1e-d036-45a2-bf00-60539701c98e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm"
Sep 11 20:21:41.154698 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:41.154605    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb\" (UID: \"cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb"
Sep 11 20:21:41.154698 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:41.154653    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcda5f1e-d036-45a2-bf00-60539701c98e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm\" (UID: \"dcda5f1e-d036-45a2-bf00-60539701c98e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm"
Sep 11 20:21:41.154933 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:41.154714    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8zxw\" (UniqueName: \"kubernetes.io/projected/cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e-kube-api-access-f8zxw\") pod \"payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb\" (UID: \"cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb"
Sep 11 20:21:41.154933 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:41.154848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv9f5\" (UniqueName: \"kubernetes.io/projected/dcda5f1e-d036-45a2-bf00-60539701c98e-kube-api-access-gv9f5\") pod \"payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm\" (UID: \"dcda5f1e-d036-45a2-bf00-60539701c98e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm"
Sep 11 20:21:41.155418 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:41.155391    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcda5f1e-d036-45a2-bf00-60539701c98e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm\" (UID: \"dcda5f1e-d036-45a2-bf00-60539701c98e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm"
Sep 11 20:21:41.155418 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:41.155409    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb\" (UID: \"cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb"
Sep 11 20:21:41.162893 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:41.162872    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8zxw\" (UniqueName: \"kubernetes.io/projected/cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e-kube-api-access-f8zxw\") pod \"payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb\" (UID: \"cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb"
Sep 11 20:21:41.163053 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:41.163033    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv9f5\" (UniqueName: \"kubernetes.io/projected/dcda5f1e-d036-45a2-bf00-60539701c98e-kube-api-access-gv9f5\") pod \"payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm\" (UID: \"dcda5f1e-d036-45a2-bf00-60539701c98e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm"
Sep 11 20:21:41.207661 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:41.207632    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb"
Sep 11 20:21:41.232924 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:41.232897    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm"
Sep 11 20:21:41.483712 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:41.483684    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb"]
Sep 11 20:21:41.484935 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:21:41.484913    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc34adf_5bdf_44d9_a36d_7eb4c4896c3e.slice/crio-91d85c7c73a7d59efe0f9c257740a2e32fd0d50a1ed13ff250f34c372471e33e WatchSource:0}: Error finding container 91d85c7c73a7d59efe0f9c257740a2e32fd0d50a1ed13ff250f34c372471e33e: Status 404 returned error can't find the container with id 91d85c7c73a7d59efe0f9c257740a2e32fd0d50a1ed13ff250f34c372471e33e
Sep 11 20:21:41.515111 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:41.515086    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm"]
Sep 11 20:21:41.519603 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:21:41.519576    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcda5f1e_d036_45a2_bf00_60539701c98e.slice/crio-29190ae698e856cc26cab565316850413fce6d880c5f97ed4ec6e1a016f4e9da WatchSource:0}: Error finding container 29190ae698e856cc26cab565316850413fce6d880c5f97ed4ec6e1a016f4e9da: Status 404 returned error can't find the container with id 29190ae698e856cc26cab565316850413fce6d880c5f97ed4ec6e1a016f4e9da
Sep 11 20:21:42.229780 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:42.229747    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb" event={"ID":"cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e","Type":"ContainerStarted","Data":"5209a22fb72eb3e71ff774c989a4e709bb3b760315fcbb41e9cb3be583b9a211"}
Sep 11 20:21:42.230230 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:42.229804    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb" event={"ID":"cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e","Type":"ContainerStarted","Data":"91d85c7c73a7d59efe0f9c257740a2e32fd0d50a1ed13ff250f34c372471e33e"}
Sep 11 20:21:42.230230 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:42.229966    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb"
Sep 11 20:21:42.231573 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:42.231544    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm" event={"ID":"dcda5f1e-d036-45a2-bf00-60539701c98e","Type":"ContainerStarted","Data":"c5f9f737b40520cafc6ea66a89e50d758ce0f9d1a132263a5b92c4e41c18087c"}
Sep 11 20:21:42.231691 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:42.231592    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm"
Sep 11 20:21:42.231691 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:42.231616    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm" event={"ID":"dcda5f1e-d036-45a2-bf00-60539701c98e","Type":"ContainerStarted","Data":"29190ae698e856cc26cab565316850413fce6d880c5f97ed4ec6e1a016f4e9da"}
Sep 11 20:21:42.258071 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:42.258015    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb" podStartSLOduration=2.258000971 podStartE2EDuration="2.258000971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:21:42.254640747 +0000 UTC m=+1272.246803015" watchObservedRunningTime="2026-09-11 20:21:42.258000971 +0000 UTC m=+1272.250163240"
Sep 11 20:21:42.295095 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:42.295049    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm" podStartSLOduration=2.295031466 podStartE2EDuration="2.295031466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:21:42.294186745 +0000 UTC m=+1272.286349014" watchObservedRunningTime="2026-09-11 20:21:42.295031466 +0000 UTC m=+1272.287193737"
Sep 11 20:21:46.889840 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:46.889809    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9d9abb-29819295-8hrtp"]
Sep 11 20:21:46.892154 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:46.892130    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9d9abb-29819295-8hrtp"]
Sep 11 20:21:46.920775 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:46.920745    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8"]
Sep 11 20:21:46.921063 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:46.921033    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8" podUID="98d64d0d-e82c-4071-8239-4d785b143bec" containerName="payload-processing" containerID="cri-o://48ee590179c01fce2f58e8b793e50d29a2b041796e7910e19f441361fc7f370e" gracePeriod=30
Sep 11 20:21:46.928465 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:46.928433    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt"]
Sep 11 20:21:46.928713 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:46.928685    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt" podUID="34faadd9-bb65-4973-8af6-150cdbbd0287" containerName="payload-pre-processing" containerID="cri-o://a0ddd3dc64078e3cd9e152f942323782f4d999c2c238e7bfd8fba1288ec1fa28" gracePeriod=30
Sep 11 20:21:47.109747 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.109714    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9d9abb-x2nd7"]
Sep 11 20:21:47.115758 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.115731    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9d9abb-x2nd7"]
Sep 11 20:21:47.204437 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.204409    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8"
Sep 11 20:21:47.207272 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.207253    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt"
Sep 11 20:21:47.211114 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.211096    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98d64d0d-e82c-4071-8239-4d785b143bec-plugins-config-volume\") pod \"98d64d0d-e82c-4071-8239-4d785b143bec\" (UID: \"98d64d0d-e82c-4071-8239-4d785b143bec\") "
Sep 11 20:21:47.211182 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.211170    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34faadd9-bb65-4973-8af6-150cdbbd0287-plugins-config-volume\") pod \"34faadd9-bb65-4973-8af6-150cdbbd0287\" (UID: \"34faadd9-bb65-4973-8af6-150cdbbd0287\") "
Sep 11 20:21:47.211237 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.211200    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmjzg\" (UniqueName: \"kubernetes.io/projected/34faadd9-bb65-4973-8af6-150cdbbd0287-kube-api-access-bmjzg\") pod \"34faadd9-bb65-4973-8af6-150cdbbd0287\" (UID: \"34faadd9-bb65-4973-8af6-150cdbbd0287\") "
Sep 11 20:21:47.211237 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.211219    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z9sc\" (UniqueName: \"kubernetes.io/projected/98d64d0d-e82c-4071-8239-4d785b143bec-kube-api-access-6z9sc\") pod \"98d64d0d-e82c-4071-8239-4d785b143bec\" (UID: \"98d64d0d-e82c-4071-8239-4d785b143bec\") "
Sep 11 20:21:47.211520 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.211479    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98d64d0d-e82c-4071-8239-4d785b143bec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "98d64d0d-e82c-4071-8239-4d785b143bec" (UID: "98d64d0d-e82c-4071-8239-4d785b143bec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:21:47.211598 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.211533    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34faadd9-bb65-4973-8af6-150cdbbd0287-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "34faadd9-bb65-4973-8af6-150cdbbd0287" (UID: "34faadd9-bb65-4973-8af6-150cdbbd0287"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:21:47.213247 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.213226    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34faadd9-bb65-4973-8af6-150cdbbd0287-kube-api-access-bmjzg" (OuterVolumeSpecName: "kube-api-access-bmjzg") pod "34faadd9-bb65-4973-8af6-150cdbbd0287" (UID: "34faadd9-bb65-4973-8af6-150cdbbd0287"). InnerVolumeSpecName "kube-api-access-bmjzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:21:47.213339 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.213244    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d64d0d-e82c-4071-8239-4d785b143bec-kube-api-access-6z9sc" (OuterVolumeSpecName: "kube-api-access-6z9sc") pod "98d64d0d-e82c-4071-8239-4d785b143bec" (UID: "98d64d0d-e82c-4071-8239-4d785b143bec"). InnerVolumeSpecName "kube-api-access-6z9sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:21:47.263051 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.263019    2580 generic.go:356] "Generic (PLEG): container finished" podID="98d64d0d-e82c-4071-8239-4d785b143bec" containerID="48ee590179c01fce2f58e8b793e50d29a2b041796e7910e19f441361fc7f370e" exitCode=0
Sep 11 20:21:47.263219 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.263053    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8"
Sep 11 20:21:47.263219 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.263096    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8" event={"ID":"98d64d0d-e82c-4071-8239-4d785b143bec","Type":"ContainerDied","Data":"48ee590179c01fce2f58e8b793e50d29a2b041796e7910e19f441361fc7f370e"}
Sep 11 20:21:47.263219 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.263158    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8" event={"ID":"98d64d0d-e82c-4071-8239-4d785b143bec","Type":"ContainerDied","Data":"a3f256414da3fc76fc803cd45017bf438ba3608bc8e1b2e79c9b620698b3ed3e"}
Sep 11 20:21:47.263219 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.263189    2580 scope.go:117] "RemoveContainer" containerID="48ee590179c01fce2f58e8b793e50d29a2b041796e7910e19f441361fc7f370e"
Sep 11 20:21:47.264811 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.264786    2580 generic.go:356] "Generic (PLEG): container finished" podID="34faadd9-bb65-4973-8af6-150cdbbd0287" containerID="a0ddd3dc64078e3cd9e152f942323782f4d999c2c238e7bfd8fba1288ec1fa28" exitCode=0
Sep 11 20:21:47.264935 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.264818    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt"
Sep 11 20:21:47.264935 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.264845    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt" event={"ID":"34faadd9-bb65-4973-8af6-150cdbbd0287","Type":"ContainerDied","Data":"a0ddd3dc64078e3cd9e152f942323782f4d999c2c238e7bfd8fba1288ec1fa28"}
Sep 11 20:21:47.264935 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.264888    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt" event={"ID":"34faadd9-bb65-4973-8af6-150cdbbd0287","Type":"ContainerDied","Data":"f4d45ccf5a8606c4eeec487897e46c6c375901882136fa34f3472d7c992dce7b"}
Sep 11 20:21:47.274082 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.274060    2580 scope.go:117] "RemoveContainer" containerID="48ee590179c01fce2f58e8b793e50d29a2b041796e7910e19f441361fc7f370e"
Sep 11 20:21:47.274382 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:21:47.274362    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48ee590179c01fce2f58e8b793e50d29a2b041796e7910e19f441361fc7f370e\": container with ID starting with 48ee590179c01fce2f58e8b793e50d29a2b041796e7910e19f441361fc7f370e not found: ID does not exist" containerID="48ee590179c01fce2f58e8b793e50d29a2b041796e7910e19f441361fc7f370e"
Sep 11 20:21:47.274439 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.274391    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48ee590179c01fce2f58e8b793e50d29a2b041796e7910e19f441361fc7f370e"} err="failed to get container status \"48ee590179c01fce2f58e8b793e50d29a2b041796e7910e19f441361fc7f370e\": rpc error: code = NotFound desc = could not find container \"48ee590179c01fce2f58e8b793e50d29a2b041796e7910e19f441361fc7f370e\": container with ID starting with 48ee590179c01fce2f58e8b793e50d29a2b041796e7910e19f441361fc7f370e not found: ID does not exist"
Sep 11 20:21:47.274439 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.274407    2580 scope.go:117] "RemoveContainer" containerID="a0ddd3dc64078e3cd9e152f942323782f4d999c2c238e7bfd8fba1288ec1fa28"
Sep 11 20:21:47.282611 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.282592    2580 scope.go:117] "RemoveContainer" containerID="a0ddd3dc64078e3cd9e152f942323782f4d999c2c238e7bfd8fba1288ec1fa28"
Sep 11 20:21:47.282868 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:21:47.282848    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0ddd3dc64078e3cd9e152f942323782f4d999c2c238e7bfd8fba1288ec1fa28\": container with ID starting with a0ddd3dc64078e3cd9e152f942323782f4d999c2c238e7bfd8fba1288ec1fa28 not found: ID does not exist" containerID="a0ddd3dc64078e3cd9e152f942323782f4d999c2c238e7bfd8fba1288ec1fa28"
Sep 11 20:21:47.282908 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.282876    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ddd3dc64078e3cd9e152f942323782f4d999c2c238e7bfd8fba1288ec1fa28"} err="failed to get container status \"a0ddd3dc64078e3cd9e152f942323782f4d999c2c238e7bfd8fba1288ec1fa28\": rpc error: code = NotFound desc = could not find container \"a0ddd3dc64078e3cd9e152f942323782f4d999c2c238e7bfd8fba1288ec1fa28\": container with ID starting with a0ddd3dc64078e3cd9e152f942323782f4d999c2c238e7bfd8fba1288ec1fa28 not found: ID does not exist"
Sep 11 20:21:47.289921 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.289900    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8"]
Sep 11 20:21:47.295794 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.295776    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9d9abb-66d5c96cd5-w7kk8"]
Sep 11 20:21:47.306023 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.306002    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt"]
Sep 11 20:21:47.308822 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.308803    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9d9abb-66c75bc7d-wxnmt"]
Sep 11 20:21:47.311945 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.311929    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmjzg\" (UniqueName: \"kubernetes.io/projected/34faadd9-bb65-4973-8af6-150cdbbd0287-kube-api-access-bmjzg\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:47.312012 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.311948    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z9sc\" (UniqueName: \"kubernetes.io/projected/98d64d0d-e82c-4071-8239-4d785b143bec-kube-api-access-6z9sc\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:47.312012 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.311958    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98d64d0d-e82c-4071-8239-4d785b143bec-plugins-config-volume\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:47.312012 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:47.311968    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34faadd9-bb65-4973-8af6-150cdbbd0287-plugins-config-volume\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:48.441631 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:48.441603    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1030e004-bc97-46ac-8e16-1ae0b0b3021c" path="/var/lib/kubelet/pods/1030e004-bc97-46ac-8e16-1ae0b0b3021c/volumes"
Sep 11 20:21:48.441999 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:48.441976    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34faadd9-bb65-4973-8af6-150cdbbd0287" path="/var/lib/kubelet/pods/34faadd9-bb65-4973-8af6-150cdbbd0287/volumes"
Sep 11 20:21:48.442233 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:48.442220    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42b47b9a-3452-4feb-8841-f0180ea4e647" path="/var/lib/kubelet/pods/42b47b9a-3452-4feb-8841-f0180ea4e647/volumes"
Sep 11 20:21:48.442538 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:48.442526    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98d64d0d-e82c-4071-8239-4d785b143bec" path="/var/lib/kubelet/pods/98d64d0d-e82c-4071-8239-4d785b143bec/volumes"
Sep 11 20:21:49.040557 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.040480    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg" podUID="a81a2439-0fc8-4113-88f1-b5903a79cba8" containerName="main" containerID="cri-o://aba30f8251f7185ad501edfa4da44786afb1e8b6fddd434c14a9ce0d04091d70" gracePeriod=45
Sep 11 20:21:49.062985 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.062948    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn" podUID="586ad6db-0f64-4427-8e12-6e13856bea86" containerName="main" containerID="cri-o://a2eed54ead3af62886d6b4bf6cb6b621ff3508bb0eb89988f04d190b38886e59" gracePeriod=45
Sep 11 20:21:49.279603 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.279575    2580 generic.go:356] "Generic (PLEG): container finished" podID="a81a2439-0fc8-4113-88f1-b5903a79cba8" containerID="aba30f8251f7185ad501edfa4da44786afb1e8b6fddd434c14a9ce0d04091d70" exitCode=0
Sep 11 20:21:49.279763 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.279646    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg" event={"ID":"a81a2439-0fc8-4113-88f1-b5903a79cba8","Type":"ContainerDied","Data":"aba30f8251f7185ad501edfa4da44786afb1e8b6fddd434c14a9ce0d04091d70"}
Sep 11 20:21:49.281282 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.281260    2580 generic.go:356] "Generic (PLEG): container finished" podID="586ad6db-0f64-4427-8e12-6e13856bea86" containerID="a2eed54ead3af62886d6b4bf6cb6b621ff3508bb0eb89988f04d190b38886e59" exitCode=0
Sep 11 20:21:49.281391 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.281289    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn" event={"ID":"586ad6db-0f64-4427-8e12-6e13856bea86","Type":"ContainerDied","Data":"a2eed54ead3af62886d6b4bf6cb6b621ff3508bb0eb89988f04d190b38886e59"}
Sep 11 20:21:49.321242 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.321138    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:21:49.335325 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.335300    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:21:49.425210 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.425181    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a81a2439-0fc8-4113-88f1-b5903a79cba8-dshm\") pod \"a81a2439-0fc8-4113-88f1-b5903a79cba8\" (UID: \"a81a2439-0fc8-4113-88f1-b5903a79cba8\") "
Sep 11 20:21:49.425386 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.425215    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a81a2439-0fc8-4113-88f1-b5903a79cba8-tls-certs\") pod \"a81a2439-0fc8-4113-88f1-b5903a79cba8\" (UID: \"a81a2439-0fc8-4113-88f1-b5903a79cba8\") "
Sep 11 20:21:49.425386 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.425244    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/586ad6db-0f64-4427-8e12-6e13856bea86-tmp-dir\") pod \"586ad6db-0f64-4427-8e12-6e13856bea86\" (UID: \"586ad6db-0f64-4427-8e12-6e13856bea86\") "
Sep 11 20:21:49.425386 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.425272    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a81a2439-0fc8-4113-88f1-b5903a79cba8-tmp-dir\") pod \"a81a2439-0fc8-4113-88f1-b5903a79cba8\" (UID: \"a81a2439-0fc8-4113-88f1-b5903a79cba8\") "
Sep 11 20:21:49.425386 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.425305    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/586ad6db-0f64-4427-8e12-6e13856bea86-tls-certs\") pod \"586ad6db-0f64-4427-8e12-6e13856bea86\" (UID: \"586ad6db-0f64-4427-8e12-6e13856bea86\") "
Sep 11 20:21:49.425386 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.425329    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/586ad6db-0f64-4427-8e12-6e13856bea86-dshm\") pod \"586ad6db-0f64-4427-8e12-6e13856bea86\" (UID: \"586ad6db-0f64-4427-8e12-6e13856bea86\") "
Sep 11 20:21:49.425386 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.425366    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/586ad6db-0f64-4427-8e12-6e13856bea86-model-cache\") pod \"586ad6db-0f64-4427-8e12-6e13856bea86\" (UID: \"586ad6db-0f64-4427-8e12-6e13856bea86\") "
Sep 11 20:21:49.425751 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.425406    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wsxs\" (UniqueName: \"kubernetes.io/projected/a81a2439-0fc8-4113-88f1-b5903a79cba8-kube-api-access-6wsxs\") pod \"a81a2439-0fc8-4113-88f1-b5903a79cba8\" (UID: \"a81a2439-0fc8-4113-88f1-b5903a79cba8\") "
Sep 11 20:21:49.425751 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.425442    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-479n2\" (UniqueName: \"kubernetes.io/projected/586ad6db-0f64-4427-8e12-6e13856bea86-kube-api-access-479n2\") pod \"586ad6db-0f64-4427-8e12-6e13856bea86\" (UID: \"586ad6db-0f64-4427-8e12-6e13856bea86\") "
Sep 11 20:21:49.425751 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.425483    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a81a2439-0fc8-4113-88f1-b5903a79cba8-model-cache\") pod \"a81a2439-0fc8-4113-88f1-b5903a79cba8\" (UID: \"a81a2439-0fc8-4113-88f1-b5903a79cba8\") "
Sep 11 20:21:49.425751 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.425553    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586ad6db-0f64-4427-8e12-6e13856bea86-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "586ad6db-0f64-4427-8e12-6e13856bea86" (UID: "586ad6db-0f64-4427-8e12-6e13856bea86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:21:49.425751 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.425578    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/586ad6db-0f64-4427-8e12-6e13856bea86-home\") pod \"586ad6db-0f64-4427-8e12-6e13856bea86\" (UID: \"586ad6db-0f64-4427-8e12-6e13856bea86\") "
Sep 11 20:21:49.425751 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.425579    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81a2439-0fc8-4113-88f1-b5903a79cba8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a81a2439-0fc8-4113-88f1-b5903a79cba8" (UID: "a81a2439-0fc8-4113-88f1-b5903a79cba8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:21:49.426048 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.425808    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586ad6db-0f64-4427-8e12-6e13856bea86-home" (OuterVolumeSpecName: "home") pod "586ad6db-0f64-4427-8e12-6e13856bea86" (UID: "586ad6db-0f64-4427-8e12-6e13856bea86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:21:49.426048 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.425822    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586ad6db-0f64-4427-8e12-6e13856bea86-model-cache" (OuterVolumeSpecName: "model-cache") pod "586ad6db-0f64-4427-8e12-6e13856bea86" (UID: "586ad6db-0f64-4427-8e12-6e13856bea86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:21:49.426048 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.425870    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a81a2439-0fc8-4113-88f1-b5903a79cba8-home\") pod \"a81a2439-0fc8-4113-88f1-b5903a79cba8\" (UID: \"a81a2439-0fc8-4113-88f1-b5903a79cba8\") "
Sep 11 20:21:49.426048 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.426003    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81a2439-0fc8-4113-88f1-b5903a79cba8-model-cache" (OuterVolumeSpecName: "model-cache") pod "a81a2439-0fc8-4113-88f1-b5903a79cba8" (UID: "a81a2439-0fc8-4113-88f1-b5903a79cba8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:21:49.426293 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.426275    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/586ad6db-0f64-4427-8e12-6e13856bea86-home\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:49.426355 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.426323    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/586ad6db-0f64-4427-8e12-6e13856bea86-tmp-dir\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:49.426355 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.426338    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a81a2439-0fc8-4113-88f1-b5903a79cba8-tmp-dir\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:49.426355 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.426351    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/586ad6db-0f64-4427-8e12-6e13856bea86-model-cache\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:49.426487 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.426364    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a81a2439-0fc8-4113-88f1-b5903a79cba8-model-cache\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:49.426487 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.426279    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81a2439-0fc8-4113-88f1-b5903a79cba8-home" (OuterVolumeSpecName: "home") pod "a81a2439-0fc8-4113-88f1-b5903a79cba8" (UID: "a81a2439-0fc8-4113-88f1-b5903a79cba8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:21:49.427722 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.427670    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/586ad6db-0f64-4427-8e12-6e13856bea86-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "586ad6db-0f64-4427-8e12-6e13856bea86" (UID: "586ad6db-0f64-4427-8e12-6e13856bea86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:21:49.427861 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.427760    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a81a2439-0fc8-4113-88f1-b5903a79cba8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a81a2439-0fc8-4113-88f1-b5903a79cba8" (UID: "a81a2439-0fc8-4113-88f1-b5903a79cba8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:21:49.427861 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.427760    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81a2439-0fc8-4113-88f1-b5903a79cba8-dshm" (OuterVolumeSpecName: "dshm") pod "a81a2439-0fc8-4113-88f1-b5903a79cba8" (UID: "a81a2439-0fc8-4113-88f1-b5903a79cba8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:21:49.427976 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.427887    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/586ad6db-0f64-4427-8e12-6e13856bea86-kube-api-access-479n2" (OuterVolumeSpecName: "kube-api-access-479n2") pod "586ad6db-0f64-4427-8e12-6e13856bea86" (UID: "586ad6db-0f64-4427-8e12-6e13856bea86"). InnerVolumeSpecName "kube-api-access-479n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:21:49.428178 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.428150    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a81a2439-0fc8-4113-88f1-b5903a79cba8-kube-api-access-6wsxs" (OuterVolumeSpecName: "kube-api-access-6wsxs") pod "a81a2439-0fc8-4113-88f1-b5903a79cba8" (UID: "a81a2439-0fc8-4113-88f1-b5903a79cba8"). InnerVolumeSpecName "kube-api-access-6wsxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:21:49.428636 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.428619    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586ad6db-0f64-4427-8e12-6e13856bea86-dshm" (OuterVolumeSpecName: "dshm") pod "586ad6db-0f64-4427-8e12-6e13856bea86" (UID: "586ad6db-0f64-4427-8e12-6e13856bea86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:21:49.527314 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.527285    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a81a2439-0fc8-4113-88f1-b5903a79cba8-home\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:49.527314 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.527308    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a81a2439-0fc8-4113-88f1-b5903a79cba8-dshm\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:49.527697 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.527320    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a81a2439-0fc8-4113-88f1-b5903a79cba8-tls-certs\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:49.527697 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.527330    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/586ad6db-0f64-4427-8e12-6e13856bea86-tls-certs\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:49.527697 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.527337    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/586ad6db-0f64-4427-8e12-6e13856bea86-dshm\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:49.527697 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.527346    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wsxs\" (UniqueName: \"kubernetes.io/projected/a81a2439-0fc8-4113-88f1-b5903a79cba8-kube-api-access-6wsxs\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:49.527697 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:49.527354    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-479n2\" (UniqueName: \"kubernetes.io/projected/586ad6db-0f64-4427-8e12-6e13856bea86-kube-api-access-479n2\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:21:50.290106 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:50.290072    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"
Sep 11 20:21:50.290274 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:50.290078    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg" event={"ID":"a81a2439-0fc8-4113-88f1-b5903a79cba8","Type":"ContainerDied","Data":"55d3b649ec6e0cf3ed08a1575ba2ceb88e1b97762090af2b9c87ef281812a4b1"}
Sep 11 20:21:50.290274 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:50.290224    2580 scope.go:117] "RemoveContainer" containerID="aba30f8251f7185ad501edfa4da44786afb1e8b6fddd434c14a9ce0d04091d70"
Sep 11 20:21:50.291641 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:50.291619    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"
Sep 11 20:21:50.291709 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:50.291626    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn" event={"ID":"586ad6db-0f64-4427-8e12-6e13856bea86","Type":"ContainerDied","Data":"60b5d66a2fdddbb9dcc765acba2955d2d0a25e6842f2ad9d34d6e3cd2c643af7"}
Sep 11 20:21:50.301650 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:50.301632    2580 scope.go:117] "RemoveContainer" containerID="a2eed54ead3af62886d6b4bf6cb6b621ff3508bb0eb89988f04d190b38886e59"
Sep 11 20:21:50.318952 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:50.318924    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"]
Sep 11 20:21:50.322261 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:50.322172    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9d9abb/distinct-w0-9d9abb-kserve-65b94b548d-ft6bg"]
Sep 11 20:21:50.334439 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:50.334403    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"]
Sep 11 20:21:50.336424 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:50.336403    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9d9abb/simulator-w0-9d9abb-kserve-5fd4744d5b-zzthn"]
Sep 11 20:21:50.441429 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:50.441392    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="586ad6db-0f64-4427-8e12-6e13856bea86" path="/var/lib/kubelet/pods/586ad6db-0f64-4427-8e12-6e13856bea86/volumes"
Sep 11 20:21:50.441828 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:50.441814    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a81a2439-0fc8-4113-88f1-b5903a79cba8" path="/var/lib/kubelet/pods/a81a2439-0fc8-4113-88f1-b5903a79cba8/volumes"
Sep 11 20:21:53.240004 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:53.239975    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm"
Sep 11 20:21:53.240455 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:21:53.240075    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb"
Sep 11 20:22:09.094445 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.094413    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"]
Sep 11 20:22:09.094827 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.094813    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34faadd9-bb65-4973-8af6-150cdbbd0287" containerName="payload-pre-processing"
Sep 11 20:22:09.094870 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.094829    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="34faadd9-bb65-4973-8af6-150cdbbd0287" containerName="payload-pre-processing"
Sep 11 20:22:09.094870 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.094838    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a81a2439-0fc8-4113-88f1-b5903a79cba8" containerName="main"
Sep 11 20:22:09.094870 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.094843    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a81a2439-0fc8-4113-88f1-b5903a79cba8" containerName="main"
Sep 11 20:22:09.094870 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.094853    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98d64d0d-e82c-4071-8239-4d785b143bec" containerName="payload-processing"
Sep 11 20:22:09.094870 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.094858    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="98d64d0d-e82c-4071-8239-4d785b143bec" containerName="payload-processing"
Sep 11 20:22:09.094870 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.094868    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="586ad6db-0f64-4427-8e12-6e13856bea86" containerName="main"
Sep 11 20:22:09.095079 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.094873    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="586ad6db-0f64-4427-8e12-6e13856bea86" containerName="main"
Sep 11 20:22:09.095079 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.094930    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="586ad6db-0f64-4427-8e12-6e13856bea86" containerName="main"
Sep 11 20:22:09.095079 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.094937    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="98d64d0d-e82c-4071-8239-4d785b143bec" containerName="payload-processing"
Sep 11 20:22:09.095079 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.094943    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a81a2439-0fc8-4113-88f1-b5903a79cba8" containerName="main"
Sep 11 20:22:09.095079 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.094948    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="34faadd9-bb65-4973-8af6-150cdbbd0287" containerName="payload-pre-processing"
Sep 11 20:22:09.097964 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.097948    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:22:09.100608 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.100587    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-22fc84\"/\"distinct-w4-22fc84-kserve-self-signed-certs\""
Sep 11 20:22:09.110162 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.110084    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"]
Sep 11 20:22:09.205180 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.205141    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da1dd569-c513-4d89-95e4-57a8e96676cd-tls-certs\") pod \"distinct-w4-22fc84-kserve-5f8c886f48-h5ws4\" (UID: \"da1dd569-c513-4d89-95e4-57a8e96676cd\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:22:09.205180 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.205180    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k8jc\" (UniqueName: \"kubernetes.io/projected/da1dd569-c513-4d89-95e4-57a8e96676cd-kube-api-access-9k8jc\") pod \"distinct-w4-22fc84-kserve-5f8c886f48-h5ws4\" (UID: \"da1dd569-c513-4d89-95e4-57a8e96676cd\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:22:09.205402 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.205207    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da1dd569-c513-4d89-95e4-57a8e96676cd-dshm\") pod \"distinct-w4-22fc84-kserve-5f8c886f48-h5ws4\" (UID: \"da1dd569-c513-4d89-95e4-57a8e96676cd\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:22:09.205402 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.205264    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da1dd569-c513-4d89-95e4-57a8e96676cd-tmp-dir\") pod \"distinct-w4-22fc84-kserve-5f8c886f48-h5ws4\" (UID: \"da1dd569-c513-4d89-95e4-57a8e96676cd\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:22:09.205402 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.205306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da1dd569-c513-4d89-95e4-57a8e96676cd-model-cache\") pod \"distinct-w4-22fc84-kserve-5f8c886f48-h5ws4\" (UID: \"da1dd569-c513-4d89-95e4-57a8e96676cd\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:22:09.205402 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.205399    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da1dd569-c513-4d89-95e4-57a8e96676cd-home\") pod \"distinct-w4-22fc84-kserve-5f8c886f48-h5ws4\" (UID: \"da1dd569-c513-4d89-95e4-57a8e96676cd\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:22:09.306510 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.306455    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da1dd569-c513-4d89-95e4-57a8e96676cd-tls-certs\") pod \"distinct-w4-22fc84-kserve-5f8c886f48-h5ws4\" (UID: \"da1dd569-c513-4d89-95e4-57a8e96676cd\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:22:09.306714 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.306523    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k8jc\" (UniqueName: \"kubernetes.io/projected/da1dd569-c513-4d89-95e4-57a8e96676cd-kube-api-access-9k8jc\") pod \"distinct-w4-22fc84-kserve-5f8c886f48-h5ws4\" (UID: \"da1dd569-c513-4d89-95e4-57a8e96676cd\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:22:09.306714 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.306557    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da1dd569-c513-4d89-95e4-57a8e96676cd-dshm\") pod \"distinct-w4-22fc84-kserve-5f8c886f48-h5ws4\" (UID: \"da1dd569-c513-4d89-95e4-57a8e96676cd\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:22:09.306714 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.306581    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da1dd569-c513-4d89-95e4-57a8e96676cd-tmp-dir\") pod \"distinct-w4-22fc84-kserve-5f8c886f48-h5ws4\" (UID: \"da1dd569-c513-4d89-95e4-57a8e96676cd\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:22:09.306714 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.306604    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da1dd569-c513-4d89-95e4-57a8e96676cd-model-cache\") pod \"distinct-w4-22fc84-kserve-5f8c886f48-h5ws4\" (UID: \"da1dd569-c513-4d89-95e4-57a8e96676cd\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:22:09.306714 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.306707    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da1dd569-c513-4d89-95e4-57a8e96676cd-home\") pod \"distinct-w4-22fc84-kserve-5f8c886f48-h5ws4\" (UID: \"da1dd569-c513-4d89-95e4-57a8e96676cd\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:22:09.307054 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.307031    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da1dd569-c513-4d89-95e4-57a8e96676cd-tmp-dir\") pod \"distinct-w4-22fc84-kserve-5f8c886f48-h5ws4\" (UID: \"da1dd569-c513-4d89-95e4-57a8e96676cd\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:22:09.307122 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.307098    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da1dd569-c513-4d89-95e4-57a8e96676cd-model-cache\") pod \"distinct-w4-22fc84-kserve-5f8c886f48-h5ws4\" (UID: \"da1dd569-c513-4d89-95e4-57a8e96676cd\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:22:09.307165 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.307105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da1dd569-c513-4d89-95e4-57a8e96676cd-home\") pod \"distinct-w4-22fc84-kserve-5f8c886f48-h5ws4\" (UID: \"da1dd569-c513-4d89-95e4-57a8e96676cd\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:22:09.308744 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.308725    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da1dd569-c513-4d89-95e4-57a8e96676cd-dshm\") pod \"distinct-w4-22fc84-kserve-5f8c886f48-h5ws4\" (UID: \"da1dd569-c513-4d89-95e4-57a8e96676cd\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:22:09.309092 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.309073    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da1dd569-c513-4d89-95e4-57a8e96676cd-tls-certs\") pod \"distinct-w4-22fc84-kserve-5f8c886f48-h5ws4\" (UID: \"da1dd569-c513-4d89-95e4-57a8e96676cd\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:22:09.315328 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.315308    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k8jc\" (UniqueName: \"kubernetes.io/projected/da1dd569-c513-4d89-95e4-57a8e96676cd-kube-api-access-9k8jc\") pod \"distinct-w4-22fc84-kserve-5f8c886f48-h5ws4\" (UID: \"da1dd569-c513-4d89-95e4-57a8e96676cd\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:22:09.415098 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.415011    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:22:09.605134 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.605098    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:22:09.688871 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:09.688847    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"]
Sep 11 20:22:09.690744 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:22:09.690710    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda1dd569_c513_4d89_95e4_57a8e96676cd.slice/crio-d429aca07597c602a3c0f52be3090157016895b515b694b91d6e84f015b5d2a5 WatchSource:0}: Error finding container d429aca07597c602a3c0f52be3090157016895b515b694b91d6e84f015b5d2a5: Status 404 returned error can't find the container with id d429aca07597c602a3c0f52be3090157016895b515b694b91d6e84f015b5d2a5
Sep 11 20:22:10.415804 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:10.415770    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4" event={"ID":"da1dd569-c513-4d89-95e4-57a8e96676cd","Type":"ContainerStarted","Data":"33bf36efc2bf29a05f0585ba8075f75c449209fb4a57595f656191fe639379d2"}
Sep 11 20:22:10.415804 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:10.415809    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4" event={"ID":"da1dd569-c513-4d89-95e4-57a8e96676cd","Type":"ContainerStarted","Data":"d429aca07597c602a3c0f52be3090157016895b515b694b91d6e84f015b5d2a5"}
Sep 11 20:22:10.434006 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:10.433957    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4" podStartSLOduration=1.433941204 podStartE2EDuration="1.433941204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:22:10.432718307 +0000 UTC m=+1300.424880589" watchObservedRunningTime="2026-09-11 20:22:10.433941204 +0000 UTC m=+1300.426103473"
Sep 11 20:22:12.996102 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:12.996052    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d11bc81f-npd75"]
Sep 11 20:22:12.999553 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:12.999532    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d11bc81f-npd75"
Sep 11 20:22:13.001899 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:13.001876    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7dbs5\""
Sep 11 20:22:13.001899 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:13.001896    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:22:13.006897 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:13.006876    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d11bc81f-npd75"]
Sep 11 20:22:13.142628 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:13.142594    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad8e822-c1b9-4f07-b524-5335f3dcf3f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d11bc81f-npd75\" (UID: \"cad8e822-c1b9-4f07-b524-5335f3dcf3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d11bc81f-npd75"
Sep 11 20:22:13.243980 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:13.243947    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad8e822-c1b9-4f07-b524-5335f3dcf3f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d11bc81f-npd75\" (UID: \"cad8e822-c1b9-4f07-b524-5335f3dcf3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d11bc81f-npd75"
Sep 11 20:22:13.244569 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:13.244550    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad8e822-c1b9-4f07-b524-5335f3dcf3f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d11bc81f-npd75\" (UID: \"cad8e822-c1b9-4f07-b524-5335f3dcf3f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d11bc81f-npd75"
Sep 11 20:22:13.313055 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:13.312980    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d11bc81f-npd75"
Sep 11 20:22:13.588138 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:13.588087    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d11bc81f-npd75"]
Sep 11 20:22:13.590364 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:22:13.590332    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad8e822_c1b9_4f07_b524_5335f3dcf3f2.slice/crio-cb98d92fa591f0cfce2bf9613bdfbfc7580f5fa9132c846d424d7f19693142a0 WatchSource:0}: Error finding container cb98d92fa591f0cfce2bf9613bdfbfc7580f5fa9132c846d424d7f19693142a0: Status 404 returned error can't find the container with id cb98d92fa591f0cfce2bf9613bdfbfc7580f5fa9132c846d424d7f19693142a0
Sep 11 20:22:14.439846 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:14.439814    2580 generic.go:356] "Generic (PLEG): container finished" podID="cad8e822-c1b9-4f07-b524-5335f3dcf3f2" containerID="2961ad1536473b474c460ea88a4ad652b3374c941361378b8aea58d92ed12a51" exitCode=0
Sep 11 20:22:14.440850 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:14.440827    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d11bc81f-npd75" event={"ID":"cad8e822-c1b9-4f07-b524-5335f3dcf3f2","Type":"ContainerDied","Data":"2961ad1536473b474c460ea88a4ad652b3374c941361378b8aea58d92ed12a51"}
Sep 11 20:22:14.440916 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:14.440869    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d11bc81f-npd75" event={"ID":"cad8e822-c1b9-4f07-b524-5335f3dcf3f2","Type":"ContainerStarted","Data":"cb98d92fa591f0cfce2bf9613bdfbfc7580f5fa9132c846d424d7f19693142a0"}
Sep 11 20:22:15.582035 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:15.582008    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d11bc81f-npd75"
Sep 11 20:22:15.665607 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:15.665563    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad8e822-c1b9-4f07-b524-5335f3dcf3f2-maas-api-service-ca\") pod \"cad8e822-c1b9-4f07-b524-5335f3dcf3f2\" (UID: \"cad8e822-c1b9-4f07-b524-5335f3dcf3f2\") "
Sep 11 20:22:15.665902 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:15.665882    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad8e822-c1b9-4f07-b524-5335f3dcf3f2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cad8e822-c1b9-4f07-b524-5335f3dcf3f2" (UID: "cad8e822-c1b9-4f07-b524-5335f3dcf3f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:22:15.767011 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:15.766936    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad8e822-c1b9-4f07-b524-5335f3dcf3f2-maas-api-service-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:22:16.453088 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:16.453063    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d11bc81f-npd75"
Sep 11 20:22:16.453215 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:16.453095    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d11bc81f-npd75" event={"ID":"cad8e822-c1b9-4f07-b524-5335f3dcf3f2","Type":"ContainerDied","Data":"cb98d92fa591f0cfce2bf9613bdfbfc7580f5fa9132c846d424d7f19693142a0"}
Sep 11 20:22:16.453215 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:16.453139    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb98d92fa591f0cfce2bf9613bdfbfc7580f5fa9132c846d424d7f19693142a0"
Sep 11 20:22:19.415818 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:19.415777    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:22:19.415818 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:19.415825    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:22:19.428140 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:19.428113    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:22:19.480047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:19.480024    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:22:23.883431 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:23.883399    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"]
Sep 11 20:22:23.883989 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:23.883967    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cad8e822-c1b9-4f07-b524-5335f3dcf3f2" containerName="revoke-keys"
Sep 11 20:22:23.884075 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:23.883992    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cad8e822-c1b9-4f07-b524-5335f3dcf3f2" containerName="revoke-keys"
Sep 11 20:22:23.884130 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:23.884102    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cad8e822-c1b9-4f07-b524-5335f3dcf3f2" containerName="revoke-keys"
Sep 11 20:22:23.887338 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:23.887317    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:22:23.891064 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:23.891039    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-fcb0e4\"/\"openshift-service-ca.crt\""
Sep 11 20:22:23.891205 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:23.891097    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-fcb0e4\"/\"auth-test-model-fcb0e4-kserve-self-signed-certs\""
Sep 11 20:22:23.891205 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:23.891153    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-fcb0e4\"/\"kube-root-ca.crt\""
Sep 11 20:22:23.891326 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:23.891213    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-fcb0e4\"/\"default-dockercfg-95dbk\""
Sep 11 20:22:23.898208 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:23.898187    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"]
Sep 11 20:22:24.036585 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:24.036545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w757d\" (UniqueName: \"kubernetes.io/projected/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-kube-api-access-w757d\") pod \"auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr\" (UID: \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:22:24.036762 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:24.036607    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-dshm\") pod \"auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr\" (UID: \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:22:24.036762 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:24.036640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-home\") pod \"auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr\" (UID: \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:22:24.036762 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:24.036686    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-model-cache\") pod \"auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr\" (UID: \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:22:24.036762 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:24.036714    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-tmp-dir\") pod \"auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr\" (UID: \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:22:24.036951 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:24.036829    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-tls-certs\") pod \"auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr\" (UID: \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:22:24.137358 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:24.137273    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-tls-certs\") pod \"auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr\" (UID: \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:22:24.137358 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:24.137311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w757d\" (UniqueName: \"kubernetes.io/projected/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-kube-api-access-w757d\") pod \"auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr\" (UID: \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:22:24.137358 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:24.137342    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-dshm\") pod \"auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr\" (UID: \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:22:24.137669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:24.137364    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-home\") pod \"auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr\" (UID: \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:22:24.137669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:24.137395    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-model-cache\") pod \"auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr\" (UID: \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:22:24.137669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:24.137414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-tmp-dir\") pod \"auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr\" (UID: \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:22:24.137914 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:24.137890    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-tmp-dir\") pod \"auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr\" (UID: \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:22:24.138004 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:24.137929    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-model-cache\") pod \"auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr\" (UID: \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:22:24.138004 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:24.137983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-home\") pod \"auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr\" (UID: \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:22:24.139659 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:24.139635    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-dshm\") pod \"auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr\" (UID: \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:22:24.140131 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:24.140105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-tls-certs\") pod \"auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr\" (UID: \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:22:24.145885 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:24.145866    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w757d\" (UniqueName: \"kubernetes.io/projected/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-kube-api-access-w757d\") pod \"auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr\" (UID: \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:22:24.197941 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:24.197921    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:22:24.472638 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:24.472612    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"]
Sep 11 20:22:24.474775 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:22:24.474744    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7f4251b_aab5_4fb0_975b_0dd1a82f72d4.slice/crio-877f5e7c8422e0b0ed12d7b1cb1ab699d44cc2adb70b6fa351dc57f410026a65 WatchSource:0}: Error finding container 877f5e7c8422e0b0ed12d7b1cb1ab699d44cc2adb70b6fa351dc57f410026a65: Status 404 returned error can't find the container with id 877f5e7c8422e0b0ed12d7b1cb1ab699d44cc2adb70b6fa351dc57f410026a65
Sep 11 20:22:24.499545 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:24.499517    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr" event={"ID":"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4","Type":"ContainerStarted","Data":"877f5e7c8422e0b0ed12d7b1cb1ab699d44cc2adb70b6fa351dc57f410026a65"}
Sep 11 20:22:25.507200 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:25.507159    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr" event={"ID":"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4","Type":"ContainerStarted","Data":"656440125a180fab4f0338150690861e017d6b5a79a13b17e8d5f6f818468bac"}
Sep 11 20:22:25.523698 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:25.523651    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr" podStartSLOduration=2.523636123 podStartE2EDuration="2.523636123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:22:25.522317518 +0000 UTC m=+1315.514479812" watchObservedRunningTime="2026-09-11 20:22:25.523636123 +0000 UTC m=+1315.515798393"
Sep 11 20:22:26.006337 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:26.006304    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:22:28.531252 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:28.531214    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5"]
Sep 11 20:22:28.531726 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:28.531453    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5" podUID="fd7759f6-10df-4e7a-80fe-8e428f4b526a" containerName="payload-processing" containerID="cri-o://865cc99d1bf1cf63d8a84cddfa1440bdbaf008ce9a1e59a6998680f0fa5896b4" gracePeriod=30
Sep 11 20:22:28.540562 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:28.540373    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2"]
Sep 11 20:22:28.540706 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:28.540611    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2" podUID="180469fd-1676-4f54-9b1c-2eba0fc06cdc" containerName="payload-pre-processing" containerID="cri-o://2ab8e645b4b5413c00a709083150a8ef4871ee43970fd71d3be547d9612c14e3" gracePeriod=30
Sep 11 20:22:28.705430 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:28.705400    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d11bc81f-npd75"]
Sep 11 20:22:28.710897 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:28.710865    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d11bc81f-npd75"]
Sep 11 20:22:28.822975 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:28.822944    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5"
Sep 11 20:22:28.824984 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:28.824963    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2"
Sep 11 20:22:28.979544 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:28.979484    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/180469fd-1676-4f54-9b1c-2eba0fc06cdc-plugins-config-volume\") pod \"180469fd-1676-4f54-9b1c-2eba0fc06cdc\" (UID: \"180469fd-1676-4f54-9b1c-2eba0fc06cdc\") "
Sep 11 20:22:28.979698 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:28.979600    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7759f6-10df-4e7a-80fe-8e428f4b526a-plugins-config-volume\") pod \"fd7759f6-10df-4e7a-80fe-8e428f4b526a\" (UID: \"fd7759f6-10df-4e7a-80fe-8e428f4b526a\") "
Sep 11 20:22:28.979698 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:28.979665    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ddr5\" (UniqueName: \"kubernetes.io/projected/fd7759f6-10df-4e7a-80fe-8e428f4b526a-kube-api-access-2ddr5\") pod \"fd7759f6-10df-4e7a-80fe-8e428f4b526a\" (UID: \"fd7759f6-10df-4e7a-80fe-8e428f4b526a\") "
Sep 11 20:22:28.979801 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:28.979713    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pg7fr\" (UniqueName: \"kubernetes.io/projected/180469fd-1676-4f54-9b1c-2eba0fc06cdc-kube-api-access-pg7fr\") pod \"180469fd-1676-4f54-9b1c-2eba0fc06cdc\" (UID: \"180469fd-1676-4f54-9b1c-2eba0fc06cdc\") "
Sep 11 20:22:28.979961 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:28.979933    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/180469fd-1676-4f54-9b1c-2eba0fc06cdc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "180469fd-1676-4f54-9b1c-2eba0fc06cdc" (UID: "180469fd-1676-4f54-9b1c-2eba0fc06cdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:22:28.980032 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:28.980002    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd7759f6-10df-4e7a-80fe-8e428f4b526a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd7759f6-10df-4e7a-80fe-8e428f4b526a" (UID: "fd7759f6-10df-4e7a-80fe-8e428f4b526a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:22:28.981765 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:28.981737    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/180469fd-1676-4f54-9b1c-2eba0fc06cdc-kube-api-access-pg7fr" (OuterVolumeSpecName: "kube-api-access-pg7fr") pod "180469fd-1676-4f54-9b1c-2eba0fc06cdc" (UID: "180469fd-1676-4f54-9b1c-2eba0fc06cdc"). InnerVolumeSpecName "kube-api-access-pg7fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:22:28.981895 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:28.981875    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7759f6-10df-4e7a-80fe-8e428f4b526a-kube-api-access-2ddr5" (OuterVolumeSpecName: "kube-api-access-2ddr5") pod "fd7759f6-10df-4e7a-80fe-8e428f4b526a" (UID: "fd7759f6-10df-4e7a-80fe-8e428f4b526a"). InnerVolumeSpecName "kube-api-access-2ddr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:22:29.081486 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:29.081403    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ddr5\" (UniqueName: \"kubernetes.io/projected/fd7759f6-10df-4e7a-80fe-8e428f4b526a-kube-api-access-2ddr5\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:22:29.081486 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:29.081432    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg7fr\" (UniqueName: \"kubernetes.io/projected/180469fd-1676-4f54-9b1c-2eba0fc06cdc-kube-api-access-pg7fr\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:22:29.081486 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:29.081442    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/180469fd-1676-4f54-9b1c-2eba0fc06cdc-plugins-config-volume\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:22:29.081486 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:29.081451    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7759f6-10df-4e7a-80fe-8e428f4b526a-plugins-config-volume\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:22:29.532772 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:29.532737    2580 generic.go:356] "Generic (PLEG): container finished" podID="fd7759f6-10df-4e7a-80fe-8e428f4b526a" containerID="865cc99d1bf1cf63d8a84cddfa1440bdbaf008ce9a1e59a6998680f0fa5896b4" exitCode=0
Sep 11 20:22:29.533229 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:29.532782    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5"
Sep 11 20:22:29.533229 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:29.532831    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5" event={"ID":"fd7759f6-10df-4e7a-80fe-8e428f4b526a","Type":"ContainerDied","Data":"865cc99d1bf1cf63d8a84cddfa1440bdbaf008ce9a1e59a6998680f0fa5896b4"}
Sep 11 20:22:29.533229 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:29.532883    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5" event={"ID":"fd7759f6-10df-4e7a-80fe-8e428f4b526a","Type":"ContainerDied","Data":"5721fbb7e2a897088e6067645f785b877bb596805c67d682ade96de13baa1cb7"}
Sep 11 20:22:29.533229 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:29.532907    2580 scope.go:117] "RemoveContainer" containerID="865cc99d1bf1cf63d8a84cddfa1440bdbaf008ce9a1e59a6998680f0fa5896b4"
Sep 11 20:22:29.534329 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:29.534309    2580 generic.go:356] "Generic (PLEG): container finished" podID="180469fd-1676-4f54-9b1c-2eba0fc06cdc" containerID="2ab8e645b4b5413c00a709083150a8ef4871ee43970fd71d3be547d9612c14e3" exitCode=0
Sep 11 20:22:29.534408 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:29.534340    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2" event={"ID":"180469fd-1676-4f54-9b1c-2eba0fc06cdc","Type":"ContainerDied","Data":"2ab8e645b4b5413c00a709083150a8ef4871ee43970fd71d3be547d9612c14e3"}
Sep 11 20:22:29.534408 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:29.534358    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2"
Sep 11 20:22:29.534408 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:29.534404    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2" event={"ID":"180469fd-1676-4f54-9b1c-2eba0fc06cdc","Type":"ContainerDied","Data":"12acd27fb5c6495abf75c94ee487735d38e3223fdee24b7bf044ecc323567d6e"}
Sep 11 20:22:29.542913 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:29.542890    2580 scope.go:117] "RemoveContainer" containerID="865cc99d1bf1cf63d8a84cddfa1440bdbaf008ce9a1e59a6998680f0fa5896b4"
Sep 11 20:22:29.543190 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:22:29.543171    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"865cc99d1bf1cf63d8a84cddfa1440bdbaf008ce9a1e59a6998680f0fa5896b4\": container with ID starting with 865cc99d1bf1cf63d8a84cddfa1440bdbaf008ce9a1e59a6998680f0fa5896b4 not found: ID does not exist" containerID="865cc99d1bf1cf63d8a84cddfa1440bdbaf008ce9a1e59a6998680f0fa5896b4"
Sep 11 20:22:29.543257 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:29.543197    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865cc99d1bf1cf63d8a84cddfa1440bdbaf008ce9a1e59a6998680f0fa5896b4"} err="failed to get container status \"865cc99d1bf1cf63d8a84cddfa1440bdbaf008ce9a1e59a6998680f0fa5896b4\": rpc error: code = NotFound desc = could not find container \"865cc99d1bf1cf63d8a84cddfa1440bdbaf008ce9a1e59a6998680f0fa5896b4\": container with ID starting with 865cc99d1bf1cf63d8a84cddfa1440bdbaf008ce9a1e59a6998680f0fa5896b4 not found: ID does not exist"
Sep 11 20:22:29.543257 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:29.543212    2580 scope.go:117] "RemoveContainer" containerID="2ab8e645b4b5413c00a709083150a8ef4871ee43970fd71d3be547d9612c14e3"
Sep 11 20:22:29.555554 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:29.555535    2580 scope.go:117] "RemoveContainer" containerID="2ab8e645b4b5413c00a709083150a8ef4871ee43970fd71d3be547d9612c14e3"
Sep 11 20:22:29.555811 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:22:29.555794    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab8e645b4b5413c00a709083150a8ef4871ee43970fd71d3be547d9612c14e3\": container with ID starting with 2ab8e645b4b5413c00a709083150a8ef4871ee43970fd71d3be547d9612c14e3 not found: ID does not exist" containerID="2ab8e645b4b5413c00a709083150a8ef4871ee43970fd71d3be547d9612c14e3"
Sep 11 20:22:29.555875 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:29.555821    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab8e645b4b5413c00a709083150a8ef4871ee43970fd71d3be547d9612c14e3"} err="failed to get container status \"2ab8e645b4b5413c00a709083150a8ef4871ee43970fd71d3be547d9612c14e3\": rpc error: code = NotFound desc = could not find container \"2ab8e645b4b5413c00a709083150a8ef4871ee43970fd71d3be547d9612c14e3\": container with ID starting with 2ab8e645b4b5413c00a709083150a8ef4871ee43970fd71d3be547d9612c14e3 not found: ID does not exist"
Sep 11 20:22:29.567673 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:29.567646    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5"]
Sep 11 20:22:29.571646 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:29.571627    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d11bc81f-786469ff97-n2vv5"]
Sep 11 20:22:29.583208 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:29.583187    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2"]
Sep 11 20:22:29.587863 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:29.587841    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d11bc81f-75898cd89d-8r6d2"]
Sep 11 20:22:30.440918 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:30.440879    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="180469fd-1676-4f54-9b1c-2eba0fc06cdc" path="/var/lib/kubelet/pods/180469fd-1676-4f54-9b1c-2eba0fc06cdc/volumes"
Sep 11 20:22:30.441316 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:30.441295    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cad8e822-c1b9-4f07-b524-5335f3dcf3f2" path="/var/lib/kubelet/pods/cad8e822-c1b9-4f07-b524-5335f3dcf3f2/volumes"
Sep 11 20:22:30.441714 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:30.441695    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd7759f6-10df-4e7a-80fe-8e428f4b526a" path="/var/lib/kubelet/pods/fd7759f6-10df-4e7a-80fe-8e428f4b526a/volumes"
Sep 11 20:22:30.689147 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:30.689120    2580 scope.go:117] "RemoveContainer" containerID="1cca8c26c090ed614bb621b9bf993fedcc6a0aa4f1289782a67fae621f33766f"
Sep 11 20:22:34.198926 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:34.198891    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:22:34.199330 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:34.199151    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:22:34.211563 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:34.211516    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:22:34.576834 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:34.576744    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:22:36.768187 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:36.768149    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"]
Sep 11 20:22:46.811129 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:46.811091    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:22:51.787370 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:51.787325    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw" podUID="73739b33-c83d-41c0-8fc7-be15e0b122e2" containerName="main" containerID="cri-o://ab14517d6a8b8525624e13c717be348323766cb30b65af7f4d170137e8f814d8" gracePeriod=45
Sep 11 20:22:52.036734 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.036699    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:22:52.099304 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.099224    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73739b33-c83d-41c0-8fc7-be15e0b122e2-tmp-dir\") pod \"73739b33-c83d-41c0-8fc7-be15e0b122e2\" (UID: \"73739b33-c83d-41c0-8fc7-be15e0b122e2\") "
Sep 11 20:22:52.099304 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.099277    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73739b33-c83d-41c0-8fc7-be15e0b122e2-home\") pod \"73739b33-c83d-41c0-8fc7-be15e0b122e2\" (UID: \"73739b33-c83d-41c0-8fc7-be15e0b122e2\") "
Sep 11 20:22:52.099555 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.099320    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73739b33-c83d-41c0-8fc7-be15e0b122e2-dshm\") pod \"73739b33-c83d-41c0-8fc7-be15e0b122e2\" (UID: \"73739b33-c83d-41c0-8fc7-be15e0b122e2\") "
Sep 11 20:22:52.099555 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.099372    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73739b33-c83d-41c0-8fc7-be15e0b122e2-tls-certs\") pod \"73739b33-c83d-41c0-8fc7-be15e0b122e2\" (UID: \"73739b33-c83d-41c0-8fc7-be15e0b122e2\") "
Sep 11 20:22:52.099555 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.099414    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jxt6\" (UniqueName: \"kubernetes.io/projected/73739b33-c83d-41c0-8fc7-be15e0b122e2-kube-api-access-7jxt6\") pod \"73739b33-c83d-41c0-8fc7-be15e0b122e2\" (UID: \"73739b33-c83d-41c0-8fc7-be15e0b122e2\") "
Sep 11 20:22:52.099555 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.099440    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73739b33-c83d-41c0-8fc7-be15e0b122e2-model-cache\") pod \"73739b33-c83d-41c0-8fc7-be15e0b122e2\" (UID: \"73739b33-c83d-41c0-8fc7-be15e0b122e2\") "
Sep 11 20:22:52.099771 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.099574    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73739b33-c83d-41c0-8fc7-be15e0b122e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "73739b33-c83d-41c0-8fc7-be15e0b122e2" (UID: "73739b33-c83d-41c0-8fc7-be15e0b122e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:22:52.099771 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.099611    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73739b33-c83d-41c0-8fc7-be15e0b122e2-home" (OuterVolumeSpecName: "home") pod "73739b33-c83d-41c0-8fc7-be15e0b122e2" (UID: "73739b33-c83d-41c0-8fc7-be15e0b122e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:22:52.099771 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.099725    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73739b33-c83d-41c0-8fc7-be15e0b122e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "73739b33-c83d-41c0-8fc7-be15e0b122e2" (UID: "73739b33-c83d-41c0-8fc7-be15e0b122e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:22:52.099771 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.099757    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73739b33-c83d-41c0-8fc7-be15e0b122e2-tmp-dir\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:22:52.099771 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.099771    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73739b33-c83d-41c0-8fc7-be15e0b122e2-home\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:22:52.101624 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.101592    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73739b33-c83d-41c0-8fc7-be15e0b122e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "73739b33-c83d-41c0-8fc7-be15e0b122e2" (UID: "73739b33-c83d-41c0-8fc7-be15e0b122e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:22:52.101624 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.101600    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73739b33-c83d-41c0-8fc7-be15e0b122e2-dshm" (OuterVolumeSpecName: "dshm") pod "73739b33-c83d-41c0-8fc7-be15e0b122e2" (UID: "73739b33-c83d-41c0-8fc7-be15e0b122e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:22:52.101771 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.101687    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73739b33-c83d-41c0-8fc7-be15e0b122e2-kube-api-access-7jxt6" (OuterVolumeSpecName: "kube-api-access-7jxt6") pod "73739b33-c83d-41c0-8fc7-be15e0b122e2" (UID: "73739b33-c83d-41c0-8fc7-be15e0b122e2"). InnerVolumeSpecName "kube-api-access-7jxt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:22:52.200483 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.200448    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jxt6\" (UniqueName: \"kubernetes.io/projected/73739b33-c83d-41c0-8fc7-be15e0b122e2-kube-api-access-7jxt6\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:22:52.200483 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.200477    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73739b33-c83d-41c0-8fc7-be15e0b122e2-model-cache\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:22:52.200483 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.200487    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73739b33-c83d-41c0-8fc7-be15e0b122e2-dshm\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:22:52.200726 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.200513    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73739b33-c83d-41c0-8fc7-be15e0b122e2-tls-certs\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:22:52.671148 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.671113    2580 generic.go:356] "Generic (PLEG): container finished" podID="73739b33-c83d-41c0-8fc7-be15e0b122e2" containerID="ab14517d6a8b8525624e13c717be348323766cb30b65af7f4d170137e8f814d8" exitCode=0
Sep 11 20:22:52.671311 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.671157    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"
Sep 11 20:22:52.671311 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.671163    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw" event={"ID":"73739b33-c83d-41c0-8fc7-be15e0b122e2","Type":"ContainerDied","Data":"ab14517d6a8b8525624e13c717be348323766cb30b65af7f4d170137e8f814d8"}
Sep 11 20:22:52.671311 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.671195    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw" event={"ID":"73739b33-c83d-41c0-8fc7-be15e0b122e2","Type":"ContainerDied","Data":"481d0f29604a3ac80c10fa261ca5c4539256e049d6d1b4a58f873b374bf2455f"}
Sep 11 20:22:52.671311 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.671211    2580 scope.go:117] "RemoveContainer" containerID="ab14517d6a8b8525624e13c717be348323766cb30b65af7f4d170137e8f814d8"
Sep 11 20:22:52.680086 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.680068    2580 scope.go:117] "RemoveContainer" containerID="ab14517d6a8b8525624e13c717be348323766cb30b65af7f4d170137e8f814d8"
Sep 11 20:22:52.680349 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:22:52.680332    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab14517d6a8b8525624e13c717be348323766cb30b65af7f4d170137e8f814d8\": container with ID starting with ab14517d6a8b8525624e13c717be348323766cb30b65af7f4d170137e8f814d8 not found: ID does not exist" containerID="ab14517d6a8b8525624e13c717be348323766cb30b65af7f4d170137e8f814d8"
Sep 11 20:22:52.680394 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.680359    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab14517d6a8b8525624e13c717be348323766cb30b65af7f4d170137e8f814d8"} err="failed to get container status \"ab14517d6a8b8525624e13c717be348323766cb30b65af7f4d170137e8f814d8\": rpc error: code = NotFound desc = could not find container \"ab14517d6a8b8525624e13c717be348323766cb30b65af7f4d170137e8f814d8\": container with ID starting with ab14517d6a8b8525624e13c717be348323766cb30b65af7f4d170137e8f814d8 not found: ID does not exist"
Sep 11 20:22:52.688959 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.688937    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"]
Sep 11 20:22:52.692471 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:52.692450    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d11bc81f/e2e-lifecycle-model-d11bc81f-kserve-6fc798d899-ljvtw"]
Sep 11 20:22:54.440920 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:22:54.440885    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73739b33-c83d-41c0-8fc7-be15e0b122e2" path="/var/lib/kubelet/pods/73739b33-c83d-41c0-8fc7-be15e0b122e2/volumes"
Sep 11 20:23:07.605065 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.605031    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"]
Sep 11 20:23:07.605618 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.605595    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73739b33-c83d-41c0-8fc7-be15e0b122e2" containerName="main"
Sep 11 20:23:07.605677 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.605623    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="73739b33-c83d-41c0-8fc7-be15e0b122e2" containerName="main"
Sep 11 20:23:07.605677 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.605647    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="180469fd-1676-4f54-9b1c-2eba0fc06cdc" containerName="payload-pre-processing"
Sep 11 20:23:07.605677 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.605657    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="180469fd-1676-4f54-9b1c-2eba0fc06cdc" containerName="payload-pre-processing"
Sep 11 20:23:07.605677 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.605666    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd7759f6-10df-4e7a-80fe-8e428f4b526a" containerName="payload-processing"
Sep 11 20:23:07.605677 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.605672    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd7759f6-10df-4e7a-80fe-8e428f4b526a" containerName="payload-processing"
Sep 11 20:23:07.605829 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.605778    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="180469fd-1676-4f54-9b1c-2eba0fc06cdc" containerName="payload-pre-processing"
Sep 11 20:23:07.605829 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.605793    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd7759f6-10df-4e7a-80fe-8e428f4b526a" containerName="payload-processing"
Sep 11 20:23:07.605829 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.605803    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="73739b33-c83d-41c0-8fc7-be15e0b122e2" containerName="main"
Sep 11 20:23:07.608922 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.608905    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:23:07.611825 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.611805    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-22fc84\"/\"distinct-2-w4-22fc84-kserve-self-signed-certs\""
Sep 11 20:23:07.617550 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.617527    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"]
Sep 11 20:23:07.731321 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.731282    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcnb4\" (UniqueName: \"kubernetes.io/projected/6707d426-cb56-4ae4-81b1-8e9607479a43-kube-api-access-hcnb4\") pod \"distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs\" (UID: \"6707d426-cb56-4ae4-81b1-8e9607479a43\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:23:07.731518 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.731326    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6707d426-cb56-4ae4-81b1-8e9607479a43-dshm\") pod \"distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs\" (UID: \"6707d426-cb56-4ae4-81b1-8e9607479a43\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:23:07.731518 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.731358    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6707d426-cb56-4ae4-81b1-8e9607479a43-model-cache\") pod \"distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs\" (UID: \"6707d426-cb56-4ae4-81b1-8e9607479a43\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:23:07.731518 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.731413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6707d426-cb56-4ae4-81b1-8e9607479a43-tls-certs\") pod \"distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs\" (UID: \"6707d426-cb56-4ae4-81b1-8e9607479a43\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:23:07.731518 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.731446    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6707d426-cb56-4ae4-81b1-8e9607479a43-home\") pod \"distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs\" (UID: \"6707d426-cb56-4ae4-81b1-8e9607479a43\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:23:07.731518 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.731508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6707d426-cb56-4ae4-81b1-8e9607479a43-tmp-dir\") pod \"distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs\" (UID: \"6707d426-cb56-4ae4-81b1-8e9607479a43\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:23:07.832634 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.832602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcnb4\" (UniqueName: \"kubernetes.io/projected/6707d426-cb56-4ae4-81b1-8e9607479a43-kube-api-access-hcnb4\") pod \"distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs\" (UID: \"6707d426-cb56-4ae4-81b1-8e9607479a43\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:23:07.832634 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.832641    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6707d426-cb56-4ae4-81b1-8e9607479a43-dshm\") pod \"distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs\" (UID: \"6707d426-cb56-4ae4-81b1-8e9607479a43\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:23:07.832865 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.832663    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6707d426-cb56-4ae4-81b1-8e9607479a43-model-cache\") pod \"distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs\" (UID: \"6707d426-cb56-4ae4-81b1-8e9607479a43\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:23:07.832865 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.832775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6707d426-cb56-4ae4-81b1-8e9607479a43-tls-certs\") pod \"distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs\" (UID: \"6707d426-cb56-4ae4-81b1-8e9607479a43\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:23:07.832865 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.832815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6707d426-cb56-4ae4-81b1-8e9607479a43-home\") pod \"distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs\" (UID: \"6707d426-cb56-4ae4-81b1-8e9607479a43\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:23:07.833025 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.832872    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6707d426-cb56-4ae4-81b1-8e9607479a43-tmp-dir\") pod \"distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs\" (UID: \"6707d426-cb56-4ae4-81b1-8e9607479a43\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:23:07.833109 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.833086    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6707d426-cb56-4ae4-81b1-8e9607479a43-model-cache\") pod \"distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs\" (UID: \"6707d426-cb56-4ae4-81b1-8e9607479a43\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:23:07.833183 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.833124    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6707d426-cb56-4ae4-81b1-8e9607479a43-home\") pod \"distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs\" (UID: \"6707d426-cb56-4ae4-81b1-8e9607479a43\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:23:07.833231 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.833198    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6707d426-cb56-4ae4-81b1-8e9607479a43-tmp-dir\") pod \"distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs\" (UID: \"6707d426-cb56-4ae4-81b1-8e9607479a43\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:23:07.834846 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.834822    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6707d426-cb56-4ae4-81b1-8e9607479a43-dshm\") pod \"distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs\" (UID: \"6707d426-cb56-4ae4-81b1-8e9607479a43\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:23:07.835124 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.835108    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6707d426-cb56-4ae4-81b1-8e9607479a43-tls-certs\") pod \"distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs\" (UID: \"6707d426-cb56-4ae4-81b1-8e9607479a43\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:23:07.841936 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.841888    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcnb4\" (UniqueName: \"kubernetes.io/projected/6707d426-cb56-4ae4-81b1-8e9607479a43-kube-api-access-hcnb4\") pod \"distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs\" (UID: \"6707d426-cb56-4ae4-81b1-8e9607479a43\") " pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:23:07.918594 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:07.918561    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:23:08.192322 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:08.192288    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"]
Sep 11 20:23:08.193484 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:23:08.193454    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6707d426_cb56_4ae4_81b1_8e9607479a43.slice/crio-cda9bebc8e27e5f5053a180c31829eee55cd331b44ce0d1bc92ac147308f0b91 WatchSource:0}: Error finding container cda9bebc8e27e5f5053a180c31829eee55cd331b44ce0d1bc92ac147308f0b91: Status 404 returned error can't find the container with id cda9bebc8e27e5f5053a180c31829eee55cd331b44ce0d1bc92ac147308f0b91
Sep 11 20:23:08.772867 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:08.772824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs" event={"ID":"6707d426-cb56-4ae4-81b1-8e9607479a43","Type":"ContainerStarted","Data":"fa5e64b064aaca35405f6ed30cdbffe7ed61114e9fc2d307e5cfb3c98c01d07d"}
Sep 11 20:23:08.773241 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:08.772875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs" event={"ID":"6707d426-cb56-4ae4-81b1-8e9607479a43","Type":"ContainerStarted","Data":"cda9bebc8e27e5f5053a180c31829eee55cd331b44ce0d1bc92ac147308f0b91"}
Sep 11 20:23:08.792567 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:08.792517    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs" podStartSLOduration=1.792501348 podStartE2EDuration="1.792501348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:23:08.789577856 +0000 UTC m=+1358.781740127" watchObservedRunningTime="2026-09-11 20:23:08.792501348 +0000 UTC m=+1358.784663609"
Sep 11 20:23:14.311787 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.311754    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:23:14.367379 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.367347    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"]
Sep 11 20:23:14.370972 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.370948    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:23:14.375136 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.375114    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-092e1a\"/\"auth-test-model-092e1a-kserve-self-signed-certs\""
Sep 11 20:23:14.375281 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.375115    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-092e1a\"/\"default-dockercfg-2slpb\""
Sep 11 20:23:14.375281 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.375115    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-092e1a\"/\"kube-root-ca.crt\""
Sep 11 20:23:14.375281 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.375157    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-092e1a\"/\"openshift-service-ca.crt\""
Sep 11 20:23:14.382688 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.382668    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"]
Sep 11 20:23:14.497550 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.497517    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccbaa69-29e9-462f-9687-98a85b69ed26-tmp-dir\") pod \"auth-test-model-092e1a-kserve-6d74ddff75-xprqk\" (UID: \"4ccbaa69-29e9-462f-9687-98a85b69ed26\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:23:14.497550 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.497552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccbaa69-29e9-462f-9687-98a85b69ed26-dshm\") pod \"auth-test-model-092e1a-kserve-6d74ddff75-xprqk\" (UID: \"4ccbaa69-29e9-462f-9687-98a85b69ed26\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:23:14.497793 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.497568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ccbaa69-29e9-462f-9687-98a85b69ed26-home\") pod \"auth-test-model-092e1a-kserve-6d74ddff75-xprqk\" (UID: \"4ccbaa69-29e9-462f-9687-98a85b69ed26\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:23:14.497793 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.497602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccbaa69-29e9-462f-9687-98a85b69ed26-tls-certs\") pod \"auth-test-model-092e1a-kserve-6d74ddff75-xprqk\" (UID: \"4ccbaa69-29e9-462f-9687-98a85b69ed26\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:23:14.497793 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.497709    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccbaa69-29e9-462f-9687-98a85b69ed26-model-cache\") pod \"auth-test-model-092e1a-kserve-6d74ddff75-xprqk\" (UID: \"4ccbaa69-29e9-462f-9687-98a85b69ed26\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:23:14.497793 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.497754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8zqf\" (UniqueName: \"kubernetes.io/projected/4ccbaa69-29e9-462f-9687-98a85b69ed26-kube-api-access-k8zqf\") pod \"auth-test-model-092e1a-kserve-6d74ddff75-xprqk\" (UID: \"4ccbaa69-29e9-462f-9687-98a85b69ed26\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:23:14.598664 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.598550    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccbaa69-29e9-462f-9687-98a85b69ed26-tmp-dir\") pod \"auth-test-model-092e1a-kserve-6d74ddff75-xprqk\" (UID: \"4ccbaa69-29e9-462f-9687-98a85b69ed26\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:23:14.598929 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.598897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccbaa69-29e9-462f-9687-98a85b69ed26-dshm\") pod \"auth-test-model-092e1a-kserve-6d74ddff75-xprqk\" (UID: \"4ccbaa69-29e9-462f-9687-98a85b69ed26\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:23:14.599039 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.599025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ccbaa69-29e9-462f-9687-98a85b69ed26-home\") pod \"auth-test-model-092e1a-kserve-6d74ddff75-xprqk\" (UID: \"4ccbaa69-29e9-462f-9687-98a85b69ed26\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:23:14.599220 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.599191    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccbaa69-29e9-462f-9687-98a85b69ed26-tls-certs\") pod \"auth-test-model-092e1a-kserve-6d74ddff75-xprqk\" (UID: \"4ccbaa69-29e9-462f-9687-98a85b69ed26\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:23:14.599395 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.599380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccbaa69-29e9-462f-9687-98a85b69ed26-model-cache\") pod \"auth-test-model-092e1a-kserve-6d74ddff75-xprqk\" (UID: \"4ccbaa69-29e9-462f-9687-98a85b69ed26\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:23:14.599546 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.599530    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8zqf\" (UniqueName: \"kubernetes.io/projected/4ccbaa69-29e9-462f-9687-98a85b69ed26-kube-api-access-k8zqf\") pod \"auth-test-model-092e1a-kserve-6d74ddff75-xprqk\" (UID: \"4ccbaa69-29e9-462f-9687-98a85b69ed26\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:23:14.601022 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.600998    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ccbaa69-29e9-462f-9687-98a85b69ed26-home\") pod \"auth-test-model-092e1a-kserve-6d74ddff75-xprqk\" (UID: \"4ccbaa69-29e9-462f-9687-98a85b69ed26\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:23:14.601475 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.601177    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccbaa69-29e9-462f-9687-98a85b69ed26-model-cache\") pod \"auth-test-model-092e1a-kserve-6d74ddff75-xprqk\" (UID: \"4ccbaa69-29e9-462f-9687-98a85b69ed26\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:23:14.601634 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.601409    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccbaa69-29e9-462f-9687-98a85b69ed26-tmp-dir\") pod \"auth-test-model-092e1a-kserve-6d74ddff75-xprqk\" (UID: \"4ccbaa69-29e9-462f-9687-98a85b69ed26\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:23:14.604141 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.604096    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccbaa69-29e9-462f-9687-98a85b69ed26-tls-certs\") pod \"auth-test-model-092e1a-kserve-6d74ddff75-xprqk\" (UID: \"4ccbaa69-29e9-462f-9687-98a85b69ed26\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:23:14.604470 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.604429    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccbaa69-29e9-462f-9687-98a85b69ed26-dshm\") pod \"auth-test-model-092e1a-kserve-6d74ddff75-xprqk\" (UID: \"4ccbaa69-29e9-462f-9687-98a85b69ed26\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:23:14.612610 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.612487    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8zqf\" (UniqueName: \"kubernetes.io/projected/4ccbaa69-29e9-462f-9687-98a85b69ed26-kube-api-access-k8zqf\") pod \"auth-test-model-092e1a-kserve-6d74ddff75-xprqk\" (UID: \"4ccbaa69-29e9-462f-9687-98a85b69ed26\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:23:14.682326 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.682286    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:23:14.969431 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:14.969405    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"]
Sep 11 20:23:14.971774 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:23:14.971746    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ccbaa69_29e9_462f_9687_98a85b69ed26.slice/crio-d27303a847a97736831eeec50d5da5556857189b4aba7c0d3f753391fcf91f6c WatchSource:0}: Error finding container d27303a847a97736831eeec50d5da5556857189b4aba7c0d3f753391fcf91f6c: Status 404 returned error can't find the container with id d27303a847a97736831eeec50d5da5556857189b4aba7c0d3f753391fcf91f6c
Sep 11 20:23:15.820308 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:15.820268    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk" event={"ID":"4ccbaa69-29e9-462f-9687-98a85b69ed26","Type":"ContainerStarted","Data":"f557b698f9e4a4af77a625fb959fd78c504ee2ad055246d92aeac1206abe8e5e"}
Sep 11 20:23:15.820791 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:15.820318    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk" event={"ID":"4ccbaa69-29e9-462f-9687-98a85b69ed26","Type":"ContainerStarted","Data":"d27303a847a97736831eeec50d5da5556857189b4aba7c0d3f753391fcf91f6c"}
Sep 11 20:23:15.839046 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:15.838977    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk" podStartSLOduration=1.838959344 podStartE2EDuration="1.838959344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:23:15.836292178 +0000 UTC m=+1365.828454439" watchObservedRunningTime="2026-09-11 20:23:15.838959344 +0000 UTC m=+1365.831121614"
Sep 11 20:23:17.918683 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:17.918643    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:23:17.919096 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:17.918698    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:23:17.931221 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:17.931192    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:23:18.847571 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:18.847535    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:23:24.683053 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:24.683012    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:23:24.683640 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:24.683067    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:23:24.695539 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:24.695514    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:23:24.882620 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:24.882590    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:23:36.702193 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:36.702153    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:23:53.904979 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:23:53.904943    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:24:18.805422 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:18.805382    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:24:30.749932 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:30.749898    2580 scope.go:117] "RemoveContainer" containerID="a6d6f52fec18932c8d697e2d6d7a794e73304c2dad159d8a6182cfcb1837f9d9"
Sep 11 20:24:30.758657 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:30.758631    2580 scope.go:117] "RemoveContainer" containerID="ad61534e9fb337c8e61ff5a99cac7f35c02d12aecee546bed71f3fc6374a9f5a"
Sep 11 20:24:45.824165 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:45.824124    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8"]
Sep 11 20:24:45.829309 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:45.829292    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8"
Sep 11 20:24:45.831866 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:45.831843    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3faec0\""
Sep 11 20:24:45.832167 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:45.832144    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-3faec0-dockercfg-n58p2\""
Sep 11 20:24:45.838741 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:45.838711    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8"]
Sep 11 20:24:45.865400 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:45.865366    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w"]
Sep 11 20:24:45.868954 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:45.868935    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w"
Sep 11 20:24:45.883175 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:45.883149    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w"]
Sep 11 20:24:45.953252 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:45.953214    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frrgc\" (UniqueName: \"kubernetes.io/projected/e95cdb8e-5d5c-475d-aa19-d76838021842-kube-api-access-frrgc\") pod \"payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8\" (UID: \"e95cdb8e-5d5c-475d-aa19-d76838021842\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8"
Sep 11 20:24:45.953432 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:45.953268    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95cdb8e-5d5c-475d-aa19-d76838021842-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8\" (UID: \"e95cdb8e-5d5c-475d-aa19-d76838021842\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8"
Sep 11 20:24:45.953432 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:45.953286    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn4qn\" (UniqueName: \"kubernetes.io/projected/4686648d-9b0e-4d84-9a1d-eed08c0ac173-kube-api-access-hn4qn\") pod \"payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w\" (UID: \"4686648d-9b0e-4d84-9a1d-eed08c0ac173\") " pod="openshift-ingress/payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w"
Sep 11 20:24:45.953432 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:45.953371    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4686648d-9b0e-4d84-9a1d-eed08c0ac173-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w\" (UID: \"4686648d-9b0e-4d84-9a1d-eed08c0ac173\") " pod="openshift-ingress/payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w"
Sep 11 20:24:46.054120 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:46.054085    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4686648d-9b0e-4d84-9a1d-eed08c0ac173-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w\" (UID: \"4686648d-9b0e-4d84-9a1d-eed08c0ac173\") " pod="openshift-ingress/payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w"
Sep 11 20:24:46.054278 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:46.054175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frrgc\" (UniqueName: \"kubernetes.io/projected/e95cdb8e-5d5c-475d-aa19-d76838021842-kube-api-access-frrgc\") pod \"payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8\" (UID: \"e95cdb8e-5d5c-475d-aa19-d76838021842\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8"
Sep 11 20:24:46.054278 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:46.054206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95cdb8e-5d5c-475d-aa19-d76838021842-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8\" (UID: \"e95cdb8e-5d5c-475d-aa19-d76838021842\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8"
Sep 11 20:24:46.054278 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:46.054222    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn4qn\" (UniqueName: \"kubernetes.io/projected/4686648d-9b0e-4d84-9a1d-eed08c0ac173-kube-api-access-hn4qn\") pod \"payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w\" (UID: \"4686648d-9b0e-4d84-9a1d-eed08c0ac173\") " pod="openshift-ingress/payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w"
Sep 11 20:24:46.054955 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:46.054937    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4686648d-9b0e-4d84-9a1d-eed08c0ac173-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w\" (UID: \"4686648d-9b0e-4d84-9a1d-eed08c0ac173\") " pod="openshift-ingress/payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w"
Sep 11 20:24:46.055000 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:46.054935    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95cdb8e-5d5c-475d-aa19-d76838021842-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8\" (UID: \"e95cdb8e-5d5c-475d-aa19-d76838021842\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8"
Sep 11 20:24:46.062953 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:46.062933    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn4qn\" (UniqueName: \"kubernetes.io/projected/4686648d-9b0e-4d84-9a1d-eed08c0ac173-kube-api-access-hn4qn\") pod \"payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w\" (UID: \"4686648d-9b0e-4d84-9a1d-eed08c0ac173\") " pod="openshift-ingress/payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w"
Sep 11 20:24:46.063154 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:46.063130    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frrgc\" (UniqueName: \"kubernetes.io/projected/e95cdb8e-5d5c-475d-aa19-d76838021842-kube-api-access-frrgc\") pod \"payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8\" (UID: \"e95cdb8e-5d5c-475d-aa19-d76838021842\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8"
Sep 11 20:24:46.140000 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:46.139915    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8"
Sep 11 20:24:46.179154 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:46.179126    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w"
Sep 11 20:24:46.424884 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:46.424860    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8"]
Sep 11 20:24:46.426641 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:24:46.426610    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95cdb8e_5d5c_475d_aa19_d76838021842.slice/crio-79fea21a4b80e3bed5fe45d2c25274842637e234f99f3230ea21d2c62af3ac04 WatchSource:0}: Error finding container 79fea21a4b80e3bed5fe45d2c25274842637e234f99f3230ea21d2c62af3ac04: Status 404 returned error can't find the container with id 79fea21a4b80e3bed5fe45d2c25274842637e234f99f3230ea21d2c62af3ac04
Sep 11 20:24:46.469353 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:46.469323    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w"]
Sep 11 20:24:46.470398 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:24:46.470370    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4686648d_9b0e_4d84_9a1d_eed08c0ac173.slice/crio-3a3a32bcd71ab011b1937df9e79cba106ca4d893c3a03293f5b3068b1b458254 WatchSource:0}: Error finding container 3a3a32bcd71ab011b1937df9e79cba106ca4d893c3a03293f5b3068b1b458254: Status 404 returned error can't find the container with id 3a3a32bcd71ab011b1937df9e79cba106ca4d893c3a03293f5b3068b1b458254
Sep 11 20:24:47.333413 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:47.333373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8" event={"ID":"e95cdb8e-5d5c-475d-aa19-d76838021842","Type":"ContainerStarted","Data":"cbb0a63e00757080aa719dd02d298aa52077c5dcd845f6f7585fb93717766a12"}
Sep 11 20:24:47.333877 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:47.333442    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8"
Sep 11 20:24:47.333877 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:47.333469    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8" event={"ID":"e95cdb8e-5d5c-475d-aa19-d76838021842","Type":"ContainerStarted","Data":"79fea21a4b80e3bed5fe45d2c25274842637e234f99f3230ea21d2c62af3ac04"}
Sep 11 20:24:47.335163 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:47.335132    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w" event={"ID":"4686648d-9b0e-4d84-9a1d-eed08c0ac173","Type":"ContainerStarted","Data":"89a971ef2c6575d6e97a52858860eaba75a2c626758e32c3dc2663eb975ea22b"}
Sep 11 20:24:47.335307 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:47.335189    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w" event={"ID":"4686648d-9b0e-4d84-9a1d-eed08c0ac173","Type":"ContainerStarted","Data":"3a3a32bcd71ab011b1937df9e79cba106ca4d893c3a03293f5b3068b1b458254"}
Sep 11 20:24:47.335307 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:47.335220    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w"
Sep 11 20:24:47.350122 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:47.350075    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8" podStartSLOduration=2.350057581 podStartE2EDuration="2.350057581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:24:47.349312213 +0000 UTC m=+1457.341474475" watchObservedRunningTime="2026-09-11 20:24:47.350057581 +0000 UTC m=+1457.342219852"
Sep 11 20:24:47.366409 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:47.366364    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w" podStartSLOduration=2.36634878 podStartE2EDuration="2.36634878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:24:47.364926501 +0000 UTC m=+1457.357088765" watchObservedRunningTime="2026-09-11 20:24:47.36634878 +0000 UTC m=+1457.358511047"
Sep 11 20:24:49.903303 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:49.903261    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:24:58.343268 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:58.343236    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8"
Sep 11 20:24:58.343657 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:24:58.343396    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w"
Sep 11 20:25:19.121903 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:19.121865    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:25:20.827432 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:20.827402    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm"]
Sep 11 20:25:20.831723 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:20.831705    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm"
Sep 11 20:25:20.834407 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:20.834388    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b0d28c\""
Sep 11 20:25:20.834531 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:20.834458    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b0d28c-dockercfg-6nhpq\""
Sep 11 20:25:20.843971 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:20.843949    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm"]
Sep 11 20:25:20.974340 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:20.974293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-998gf\" (UniqueName: \"kubernetes.io/projected/9a802b7f-9c38-486e-978e-2a5188d7caf8-kube-api-access-998gf\") pod \"payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm\" (UID: \"9a802b7f-9c38-486e-978e-2a5188d7caf8\") " pod="openshift-ingress/payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm"
Sep 11 20:25:20.974621 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:20.974362    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a802b7f-9c38-486e-978e-2a5188d7caf8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm\" (UID: \"9a802b7f-9c38-486e-978e-2a5188d7caf8\") " pod="openshift-ingress/payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm"
Sep 11 20:25:21.075839 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:21.075792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-998gf\" (UniqueName: \"kubernetes.io/projected/9a802b7f-9c38-486e-978e-2a5188d7caf8-kube-api-access-998gf\") pod \"payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm\" (UID: \"9a802b7f-9c38-486e-978e-2a5188d7caf8\") " pod="openshift-ingress/payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm"
Sep 11 20:25:21.076015 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:21.075867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a802b7f-9c38-486e-978e-2a5188d7caf8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm\" (UID: \"9a802b7f-9c38-486e-978e-2a5188d7caf8\") " pod="openshift-ingress/payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm"
Sep 11 20:25:21.076575 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:21.076557    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a802b7f-9c38-486e-978e-2a5188d7caf8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm\" (UID: \"9a802b7f-9c38-486e-978e-2a5188d7caf8\") " pod="openshift-ingress/payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm"
Sep 11 20:25:21.088212 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:21.088146    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-998gf\" (UniqueName: \"kubernetes.io/projected/9a802b7f-9c38-486e-978e-2a5188d7caf8-kube-api-access-998gf\") pod \"payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm\" (UID: \"9a802b7f-9c38-486e-978e-2a5188d7caf8\") " pod="openshift-ingress/payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm"
Sep 11 20:25:21.144134 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:21.144104    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm"
Sep 11 20:25:21.418572 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:21.418541    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm"]
Sep 11 20:25:21.420723 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:25:21.420696    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a802b7f_9c38_486e_978e_2a5188d7caf8.slice/crio-978170c085c555b7d3078d9f77bf0f04de63f84e3dcefdf049394cecea7b1420 WatchSource:0}: Error finding container 978170c085c555b7d3078d9f77bf0f04de63f84e3dcefdf049394cecea7b1420: Status 404 returned error can't find the container with id 978170c085c555b7d3078d9f77bf0f04de63f84e3dcefdf049394cecea7b1420
Sep 11 20:25:21.530254 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:21.530223    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm" event={"ID":"9a802b7f-9c38-486e-978e-2a5188d7caf8","Type":"ContainerStarted","Data":"2512ca55dcd7d66bec84db8d486fdf472edddabba57a7b28ff9fcb05ea7be866"}
Sep 11 20:25:21.530444 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:21.530274    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm" event={"ID":"9a802b7f-9c38-486e-978e-2a5188d7caf8","Type":"ContainerStarted","Data":"978170c085c555b7d3078d9f77bf0f04de63f84e3dcefdf049394cecea7b1420"}
Sep 11 20:25:21.530444 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:21.530379    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm"
Sep 11 20:25:21.547474 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:21.547416    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm" podStartSLOduration=1.547395271 podStartE2EDuration="1.547395271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:25:21.545997528 +0000 UTC m=+1491.538159799" watchObservedRunningTime="2026-09-11 20:25:21.547395271 +0000 UTC m=+1491.539557541"
Sep 11 20:25:32.538251 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:32.538217    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm"
Sep 11 20:25:41.854117 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:41.854081    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-22fc84-42ccd"]
Sep 11 20:25:41.857730 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:41.857711    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-22fc84-42ccd"
Sep 11 20:25:41.860502 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:41.860467    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:25:41.860596 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:41.860467    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7dbs5\""
Sep 11 20:25:41.866086 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:41.866064    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-22fc84-42ccd"]
Sep 11 20:25:41.871830 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:41.871801    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3530077-5092-45e4-b1c6-33c9ee6ff373-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-22fc84-42ccd\" (UID: \"a3530077-5092-45e4-b1c6-33c9ee6ff373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-22fc84-42ccd"
Sep 11 20:25:41.972902 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:41.972860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3530077-5092-45e4-b1c6-33c9ee6ff373-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-22fc84-42ccd\" (UID: \"a3530077-5092-45e4-b1c6-33c9ee6ff373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-22fc84-42ccd"
Sep 11 20:25:41.973507 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:41.973472    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3530077-5092-45e4-b1c6-33c9ee6ff373-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-22fc84-42ccd\" (UID: \"a3530077-5092-45e4-b1c6-33c9ee6ff373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-22fc84-42ccd"
Sep 11 20:25:42.168991 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:42.168893    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-22fc84-42ccd"
Sep 11 20:25:42.446910 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:42.446886    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-22fc84-42ccd"]
Sep 11 20:25:42.448536 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:25:42.448506    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3530077_5092_45e4_b1c6_33c9ee6ff373.slice/crio-466042bcdf40332eafd8191137610bdb51ced2a740238f3ceec29ebcbc2b92d8 WatchSource:0}: Error finding container 466042bcdf40332eafd8191137610bdb51ced2a740238f3ceec29ebcbc2b92d8: Status 404 returned error can't find the container with id 466042bcdf40332eafd8191137610bdb51ced2a740238f3ceec29ebcbc2b92d8
Sep 11 20:25:42.658874 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:42.658849    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-22fc84-42ccd" event={"ID":"a3530077-5092-45e4-b1c6-33c9ee6ff373","Type":"ContainerStarted","Data":"466042bcdf40332eafd8191137610bdb51ced2a740238f3ceec29ebcbc2b92d8"}
Sep 11 20:25:43.666353 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:43.666321    2580 generic.go:356] "Generic (PLEG): container finished" podID="a3530077-5092-45e4-b1c6-33c9ee6ff373" containerID="814c05b786782253bbae55729a0de152e081c1ea92ec30006d6b62f466d7adb8" exitCode=0
Sep 11 20:25:43.666772 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:43.666413    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-22fc84-42ccd" event={"ID":"a3530077-5092-45e4-b1c6-33c9ee6ff373","Type":"ContainerDied","Data":"814c05b786782253bbae55729a0de152e081c1ea92ec30006d6b62f466d7adb8"}
Sep 11 20:25:44.808428 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:44.808405    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-22fc84-42ccd"
Sep 11 20:25:44.894374 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:44.894346    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3530077-5092-45e4-b1c6-33c9ee6ff373-maas-api-service-ca\") pod \"a3530077-5092-45e4-b1c6-33c9ee6ff373\" (UID: \"a3530077-5092-45e4-b1c6-33c9ee6ff373\") "
Sep 11 20:25:44.894728 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:44.894704    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3530077-5092-45e4-b1c6-33c9ee6ff373-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a3530077-5092-45e4-b1c6-33c9ee6ff373" (UID: "a3530077-5092-45e4-b1c6-33c9ee6ff373"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:25:44.995427 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:44.995394    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3530077-5092-45e4-b1c6-33c9ee6ff373-maas-api-service-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:25:45.678909 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:45.678878    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-22fc84-42ccd"
Sep 11 20:25:45.679104 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:45.678930    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-22fc84-42ccd" event={"ID":"a3530077-5092-45e4-b1c6-33c9ee6ff373","Type":"ContainerDied","Data":"466042bcdf40332eafd8191137610bdb51ced2a740238f3ceec29ebcbc2b92d8"}
Sep 11 20:25:45.679104 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:45.678983    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="466042bcdf40332eafd8191137610bdb51ced2a740238f3ceec29ebcbc2b92d8"
Sep 11 20:25:46.722397 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:46.722362    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"]
Sep 11 20:25:46.728595 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:46.728564    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"]
Sep 11 20:25:46.745141 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:46.745109    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"]
Sep 11 20:25:47.762035 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:47.761998    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3faec0-7zc88"]
Sep 11 20:25:47.762517 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:47.762383    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3530077-5092-45e4-b1c6-33c9ee6ff373" containerName="revoke-keys"
Sep 11 20:25:47.762517 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:47.762395    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3530077-5092-45e4-b1c6-33c9ee6ff373" containerName="revoke-keys"
Sep 11 20:25:47.762517 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:47.762481    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3530077-5092-45e4-b1c6-33c9ee6ff373" containerName="revoke-keys"
Sep 11 20:25:47.765753 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:47.765736    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3faec0-7zc88"
Sep 11 20:25:47.768475 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:47.768446    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:25:47.768630 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:47.768608    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7dbs5\""
Sep 11 20:25:47.775024 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:47.775001    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3faec0-7zc88"]
Sep 11 20:25:47.821955 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:47.821925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e3e043-ad29-497f-886c-85d232dd6aaa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3faec0-7zc88\" (UID: \"66e3e043-ad29-497f-886c-85d232dd6aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3faec0-7zc88"
Sep 11 20:25:47.923330 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:47.923289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e3e043-ad29-497f-886c-85d232dd6aaa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3faec0-7zc88\" (UID: \"66e3e043-ad29-497f-886c-85d232dd6aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3faec0-7zc88"
Sep 11 20:25:47.923953 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:47.923932    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e3e043-ad29-497f-886c-85d232dd6aaa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3faec0-7zc88\" (UID: \"66e3e043-ad29-497f-886c-85d232dd6aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3faec0-7zc88"
Sep 11 20:25:48.077852 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:48.077767    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3faec0-7zc88"
Sep 11 20:25:48.224616 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:48.224573    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:25:48.352350 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:48.352327    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3faec0-7zc88"]
Sep 11 20:25:48.354284 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:25:48.354256    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e3e043_ad29_497f_886c_85d232dd6aaa.slice/crio-2bda1e8c52ac359a96bc0e96293ed24b0c6153d8210f973ffd8c5415f5af887b WatchSource:0}: Error finding container 2bda1e8c52ac359a96bc0e96293ed24b0c6153d8210f973ffd8c5415f5af887b: Status 404 returned error can't find the container with id 2bda1e8c52ac359a96bc0e96293ed24b0c6153d8210f973ffd8c5415f5af887b
Sep 11 20:25:48.698195 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:48.698155    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3faec0-7zc88" event={"ID":"66e3e043-ad29-497f-886c-85d232dd6aaa","Type":"ContainerStarted","Data":"b9ee65038617b4117506f489c4e682e583b3b6a17f2fac2370d96f17e8415def"}
Sep 11 20:25:48.698379 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:48.698228    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3faec0-7zc88" event={"ID":"66e3e043-ad29-497f-886c-85d232dd6aaa","Type":"ContainerStarted","Data":"2bda1e8c52ac359a96bc0e96293ed24b0c6153d8210f973ffd8c5415f5af887b"}
Sep 11 20:25:48.716121 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:48.716071    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3faec0-7zc88" podStartSLOduration=1.716055877 podStartE2EDuration="1.716055877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:25:48.714072615 +0000 UTC m=+1518.706234897" watchObservedRunningTime="2026-09-11 20:25:48.716055877 +0000 UTC m=+1518.708218146"
Sep 11 20:25:49.705741 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:49.705705    2580 generic.go:356] "Generic (PLEG): container finished" podID="66e3e043-ad29-497f-886c-85d232dd6aaa" containerID="b9ee65038617b4117506f489c4e682e583b3b6a17f2fac2370d96f17e8415def" exitCode=0
Sep 11 20:25:49.706130 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:49.705745    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3faec0-7zc88" event={"ID":"66e3e043-ad29-497f-886c-85d232dd6aaa","Type":"ContainerDied","Data":"b9ee65038617b4117506f489c4e682e583b3b6a17f2fac2370d96f17e8415def"}
Sep 11 20:25:50.846730 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:50.846705    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3faec0-7zc88"
Sep 11 20:25:50.950203 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:50.950179    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e3e043-ad29-497f-886c-85d232dd6aaa-maas-api-service-ca\") pod \"66e3e043-ad29-497f-886c-85d232dd6aaa\" (UID: \"66e3e043-ad29-497f-886c-85d232dd6aaa\") "
Sep 11 20:25:50.953186 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:50.950516    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66e3e043-ad29-497f-886c-85d232dd6aaa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "66e3e043-ad29-497f-886c-85d232dd6aaa" (UID: "66e3e043-ad29-497f-886c-85d232dd6aaa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:25:51.051902 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:51.051821    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e3e043-ad29-497f-886c-85d232dd6aaa-maas-api-service-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:25:51.718095 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:51.718056    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3faec0-7zc88"
Sep 11 20:25:51.718095 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:51.718083    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3faec0-7zc88" event={"ID":"66e3e043-ad29-497f-886c-85d232dd6aaa","Type":"ContainerDied","Data":"2bda1e8c52ac359a96bc0e96293ed24b0c6153d8210f973ffd8c5415f5af887b"}
Sep 11 20:25:51.718301 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:25:51.718141    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bda1e8c52ac359a96bc0e96293ed24b0c6153d8210f973ffd8c5415f5af887b"
Sep 11 20:26:00.249084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.248998    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg"]
Sep 11 20:26:00.249486 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.249225    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg" podUID="d943c471-9182-4add-a825-58deb30b3816" containerName="payload-processing" containerID="cri-o://a0afecef9b4ea4f5f28969b7cebf6cc6ddb0645e5f8da6b01a74d3a76e1e31b6" gracePeriod=30
Sep 11 20:26:00.259237 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.259210    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl"]
Sep 11 20:26:00.259641 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.259607    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl" podUID="6d36169a-7242-4916-9307-3deee76d24b1" containerName="payload-pre-processing" containerID="cri-o://e9a266b3e511b08bd9882b7f6b563c475af84d80024d2bce7f1695ce586c28f4" gracePeriod=30
Sep 11 20:26:00.462780 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.462745    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-22fc84-42ccd"]
Sep 11 20:26:00.468132 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.468098    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-22fc84-42ccd"]
Sep 11 20:26:00.531647 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.531628    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg"
Sep 11 20:26:00.534735 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.534715    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl"
Sep 11 20:26:00.645605 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.645575    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d36169a-7242-4916-9307-3deee76d24b1-plugins-config-volume\") pod \"6d36169a-7242-4916-9307-3deee76d24b1\" (UID: \"6d36169a-7242-4916-9307-3deee76d24b1\") "
Sep 11 20:26:00.645769 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.645636    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d943c471-9182-4add-a825-58deb30b3816-plugins-config-volume\") pod \"d943c471-9182-4add-a825-58deb30b3816\" (UID: \"d943c471-9182-4add-a825-58deb30b3816\") "
Sep 11 20:26:00.645769 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.645659    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgvhp\" (UniqueName: \"kubernetes.io/projected/6d36169a-7242-4916-9307-3deee76d24b1-kube-api-access-kgvhp\") pod \"6d36169a-7242-4916-9307-3deee76d24b1\" (UID: \"6d36169a-7242-4916-9307-3deee76d24b1\") "
Sep 11 20:26:00.645769 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.645709    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7vcn\" (UniqueName: \"kubernetes.io/projected/d943c471-9182-4add-a825-58deb30b3816-kube-api-access-v7vcn\") pod \"d943c471-9182-4add-a825-58deb30b3816\" (UID: \"d943c471-9182-4add-a825-58deb30b3816\") "
Sep 11 20:26:00.646064 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.646029    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d943c471-9182-4add-a825-58deb30b3816-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d943c471-9182-4add-a825-58deb30b3816" (UID: "d943c471-9182-4add-a825-58deb30b3816"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:26:00.646064 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.646028    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d36169a-7242-4916-9307-3deee76d24b1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6d36169a-7242-4916-9307-3deee76d24b1" (UID: "6d36169a-7242-4916-9307-3deee76d24b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:26:00.647787 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.647752    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d943c471-9182-4add-a825-58deb30b3816-kube-api-access-v7vcn" (OuterVolumeSpecName: "kube-api-access-v7vcn") pod "d943c471-9182-4add-a825-58deb30b3816" (UID: "d943c471-9182-4add-a825-58deb30b3816"). InnerVolumeSpecName "kube-api-access-v7vcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:26:00.647949 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.647924    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d36169a-7242-4916-9307-3deee76d24b1-kube-api-access-kgvhp" (OuterVolumeSpecName: "kube-api-access-kgvhp") pod "6d36169a-7242-4916-9307-3deee76d24b1" (UID: "6d36169a-7242-4916-9307-3deee76d24b1"). InnerVolumeSpecName "kube-api-access-kgvhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:26:00.747352 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.747322    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d943c471-9182-4add-a825-58deb30b3816-plugins-config-volume\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:00.747352 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.747350    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgvhp\" (UniqueName: \"kubernetes.io/projected/6d36169a-7242-4916-9307-3deee76d24b1-kube-api-access-kgvhp\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:00.747352 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.747360    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7vcn\" (UniqueName: \"kubernetes.io/projected/d943c471-9182-4add-a825-58deb30b3816-kube-api-access-v7vcn\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:00.747609 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.747368    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d36169a-7242-4916-9307-3deee76d24b1-plugins-config-volume\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:00.774227 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.774137    2580 generic.go:356] "Generic (PLEG): container finished" podID="6d36169a-7242-4916-9307-3deee76d24b1" containerID="e9a266b3e511b08bd9882b7f6b563c475af84d80024d2bce7f1695ce586c28f4" exitCode=0
Sep 11 20:26:00.774397 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.774266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl" event={"ID":"6d36169a-7242-4916-9307-3deee76d24b1","Type":"ContainerDied","Data":"e9a266b3e511b08bd9882b7f6b563c475af84d80024d2bce7f1695ce586c28f4"}
Sep 11 20:26:00.774397 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.774359    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl" event={"ID":"6d36169a-7242-4916-9307-3deee76d24b1","Type":"ContainerDied","Data":"a9b8d701479e7898e7df9dd072a910582366e7bc800303e2cf3807ce535cc0d9"}
Sep 11 20:26:00.774532 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.774396    2580 scope.go:117] "RemoveContainer" containerID="e9a266b3e511b08bd9882b7f6b563c475af84d80024d2bce7f1695ce586c28f4"
Sep 11 20:26:00.774635 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.774619    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl"
Sep 11 20:26:00.779861 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.779817    2580 generic.go:356] "Generic (PLEG): container finished" podID="d943c471-9182-4add-a825-58deb30b3816" containerID="a0afecef9b4ea4f5f28969b7cebf6cc6ddb0645e5f8da6b01a74d3a76e1e31b6" exitCode=0
Sep 11 20:26:00.780001 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.779870    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg"
Sep 11 20:26:00.780001 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.779875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg" event={"ID":"d943c471-9182-4add-a825-58deb30b3816","Type":"ContainerDied","Data":"a0afecef9b4ea4f5f28969b7cebf6cc6ddb0645e5f8da6b01a74d3a76e1e31b6"}
Sep 11 20:26:00.780001 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.779995    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg" event={"ID":"d943c471-9182-4add-a825-58deb30b3816","Type":"ContainerDied","Data":"8320782a0b248446f8e46d11cee8b7298c0bd7659f855b035a805b1fb05313da"}
Sep 11 20:26:00.789524 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.789481    2580 scope.go:117] "RemoveContainer" containerID="e9a266b3e511b08bd9882b7f6b563c475af84d80024d2bce7f1695ce586c28f4"
Sep 11 20:26:00.789810 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:26:00.789788    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9a266b3e511b08bd9882b7f6b563c475af84d80024d2bce7f1695ce586c28f4\": container with ID starting with e9a266b3e511b08bd9882b7f6b563c475af84d80024d2bce7f1695ce586c28f4 not found: ID does not exist" containerID="e9a266b3e511b08bd9882b7f6b563c475af84d80024d2bce7f1695ce586c28f4"
Sep 11 20:26:00.789892 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.789819    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a266b3e511b08bd9882b7f6b563c475af84d80024d2bce7f1695ce586c28f4"} err="failed to get container status \"e9a266b3e511b08bd9882b7f6b563c475af84d80024d2bce7f1695ce586c28f4\": rpc error: code = NotFound desc = could not find container \"e9a266b3e511b08bd9882b7f6b563c475af84d80024d2bce7f1695ce586c28f4\": container with ID starting with e9a266b3e511b08bd9882b7f6b563c475af84d80024d2bce7f1695ce586c28f4 not found: ID does not exist"
Sep 11 20:26:00.789892 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.789842    2580 scope.go:117] "RemoveContainer" containerID="a0afecef9b4ea4f5f28969b7cebf6cc6ddb0645e5f8da6b01a74d3a76e1e31b6"
Sep 11 20:26:00.798191 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.798174    2580 scope.go:117] "RemoveContainer" containerID="a0afecef9b4ea4f5f28969b7cebf6cc6ddb0645e5f8da6b01a74d3a76e1e31b6"
Sep 11 20:26:00.798461 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:26:00.798443    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0afecef9b4ea4f5f28969b7cebf6cc6ddb0645e5f8da6b01a74d3a76e1e31b6\": container with ID starting with a0afecef9b4ea4f5f28969b7cebf6cc6ddb0645e5f8da6b01a74d3a76e1e31b6 not found: ID does not exist" containerID="a0afecef9b4ea4f5f28969b7cebf6cc6ddb0645e5f8da6b01a74d3a76e1e31b6"
Sep 11 20:26:00.798545 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.798467    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0afecef9b4ea4f5f28969b7cebf6cc6ddb0645e5f8da6b01a74d3a76e1e31b6"} err="failed to get container status \"a0afecef9b4ea4f5f28969b7cebf6cc6ddb0645e5f8da6b01a74d3a76e1e31b6\": rpc error: code = NotFound desc = could not find container \"a0afecef9b4ea4f5f28969b7cebf6cc6ddb0645e5f8da6b01a74d3a76e1e31b6\": container with ID starting with a0afecef9b4ea4f5f28969b7cebf6cc6ddb0645e5f8da6b01a74d3a76e1e31b6 not found: ID does not exist"
Sep 11 20:26:00.805711 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.805691    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl"]
Sep 11 20:26:00.808399 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.808375    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-22fc84-74bb786798-wt6jl"]
Sep 11 20:26:00.821473 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.821451    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg"]
Sep 11 20:26:00.834044 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:00.834020    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-22fc84-bcc75d545-l9hvg"]
Sep 11 20:26:01.743017 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:01.742978    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs" podUID="6707d426-cb56-4ae4-81b1-8e9607479a43" containerName="main" containerID="cri-o://fa5e64b064aaca35405f6ed30cdbffe7ed61114e9fc2d307e5cfb3c98c01d07d" gracePeriod=45
Sep 11 20:26:01.749055 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:01.749027    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4" podUID="da1dd569-c513-4d89-95e4-57a8e96676cd" containerName="main" containerID="cri-o://33bf36efc2bf29a05f0585ba8075f75c449209fb4a57595f656191fe639379d2" gracePeriod=45
Sep 11 20:26:01.769905 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:01.769873    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs" podUID="03041350-1de6-4473-ad4c-be9e6ea72bca" containerName="main" containerID="cri-o://c32758a94a5fb446c91fbc3bc457ea27a7be613b764df3c4a9b9f4fb7c8b7aca" gracePeriod=45
Sep 11 20:26:02.051998 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.051974    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:26:02.055639 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.055617    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:26:02.058531 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.058514    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:26:02.159558 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.159523    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9k8jc\" (UniqueName: \"kubernetes.io/projected/da1dd569-c513-4d89-95e4-57a8e96676cd-kube-api-access-9k8jc\") pod \"da1dd569-c513-4d89-95e4-57a8e96676cd\" (UID: \"da1dd569-c513-4d89-95e4-57a8e96676cd\") "
Sep 11 20:26:02.159763 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.159567    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6707d426-cb56-4ae4-81b1-8e9607479a43-home\") pod \"6707d426-cb56-4ae4-81b1-8e9607479a43\" (UID: \"6707d426-cb56-4ae4-81b1-8e9607479a43\") "
Sep 11 20:26:02.159763 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.159604    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03041350-1de6-4473-ad4c-be9e6ea72bca-model-cache\") pod \"03041350-1de6-4473-ad4c-be9e6ea72bca\" (UID: \"03041350-1de6-4473-ad4c-be9e6ea72bca\") "
Sep 11 20:26:02.159763 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.159619    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6707d426-cb56-4ae4-81b1-8e9607479a43-tmp-dir\") pod \"6707d426-cb56-4ae4-81b1-8e9607479a43\" (UID: \"6707d426-cb56-4ae4-81b1-8e9607479a43\") "
Sep 11 20:26:02.159763 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.159659    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03041350-1de6-4473-ad4c-be9e6ea72bca-tmp-dir\") pod \"03041350-1de6-4473-ad4c-be9e6ea72bca\" (UID: \"03041350-1de6-4473-ad4c-be9e6ea72bca\") "
Sep 11 20:26:02.159763 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.159685    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da1dd569-c513-4d89-95e4-57a8e96676cd-model-cache\") pod \"da1dd569-c513-4d89-95e4-57a8e96676cd\" (UID: \"da1dd569-c513-4d89-95e4-57a8e96676cd\") "
Sep 11 20:26:02.159763 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.159708    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03041350-1de6-4473-ad4c-be9e6ea72bca-dshm\") pod \"03041350-1de6-4473-ad4c-be9e6ea72bca\" (UID: \"03041350-1de6-4473-ad4c-be9e6ea72bca\") "
Sep 11 20:26:02.159763 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.159731    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6707d426-cb56-4ae4-81b1-8e9607479a43-dshm\") pod \"6707d426-cb56-4ae4-81b1-8e9607479a43\" (UID: \"6707d426-cb56-4ae4-81b1-8e9607479a43\") "
Sep 11 20:26:02.159763 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.159766    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03041350-1de6-4473-ad4c-be9e6ea72bca-home\") pod \"03041350-1de6-4473-ad4c-be9e6ea72bca\" (UID: \"03041350-1de6-4473-ad4c-be9e6ea72bca\") "
Sep 11 20:26:02.160180 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.159807    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6707d426-cb56-4ae4-81b1-8e9607479a43-model-cache\") pod \"6707d426-cb56-4ae4-81b1-8e9607479a43\" (UID: \"6707d426-cb56-4ae4-81b1-8e9607479a43\") "
Sep 11 20:26:02.160180 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.159859    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnnzc\" (UniqueName: \"kubernetes.io/projected/03041350-1de6-4473-ad4c-be9e6ea72bca-kube-api-access-fnnzc\") pod \"03041350-1de6-4473-ad4c-be9e6ea72bca\" (UID: \"03041350-1de6-4473-ad4c-be9e6ea72bca\") "
Sep 11 20:26:02.160180 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.159892    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcnb4\" (UniqueName: \"kubernetes.io/projected/6707d426-cb56-4ae4-81b1-8e9607479a43-kube-api-access-hcnb4\") pod \"6707d426-cb56-4ae4-81b1-8e9607479a43\" (UID: \"6707d426-cb56-4ae4-81b1-8e9607479a43\") "
Sep 11 20:26:02.160180 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.159920    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da1dd569-c513-4d89-95e4-57a8e96676cd-tls-certs\") pod \"da1dd569-c513-4d89-95e4-57a8e96676cd\" (UID: \"da1dd569-c513-4d89-95e4-57a8e96676cd\") "
Sep 11 20:26:02.160180 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.159881    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03041350-1de6-4473-ad4c-be9e6ea72bca-model-cache" (OuterVolumeSpecName: "model-cache") pod "03041350-1de6-4473-ad4c-be9e6ea72bca" (UID: "03041350-1de6-4473-ad4c-be9e6ea72bca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:26:02.160180 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.159956    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da1dd569-c513-4d89-95e4-57a8e96676cd-home\") pod \"da1dd569-c513-4d89-95e4-57a8e96676cd\" (UID: \"da1dd569-c513-4d89-95e4-57a8e96676cd\") "
Sep 11 20:26:02.160180 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.159986    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da1dd569-c513-4d89-95e4-57a8e96676cd-tmp-dir\") pod \"da1dd569-c513-4d89-95e4-57a8e96676cd\" (UID: \"da1dd569-c513-4d89-95e4-57a8e96676cd\") "
Sep 11 20:26:02.160180 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.160013    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6707d426-cb56-4ae4-81b1-8e9607479a43-tls-certs\") pod \"6707d426-cb56-4ae4-81b1-8e9607479a43\" (UID: \"6707d426-cb56-4ae4-81b1-8e9607479a43\") "
Sep 11 20:26:02.160180 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.160054    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da1dd569-c513-4d89-95e4-57a8e96676cd-dshm\") pod \"da1dd569-c513-4d89-95e4-57a8e96676cd\" (UID: \"da1dd569-c513-4d89-95e4-57a8e96676cd\") "
Sep 11 20:26:02.160180 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.160088    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03041350-1de6-4473-ad4c-be9e6ea72bca-tls-certs\") pod \"03041350-1de6-4473-ad4c-be9e6ea72bca\" (UID: \"03041350-1de6-4473-ad4c-be9e6ea72bca\") "
Sep 11 20:26:02.160180 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.160097    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03041350-1de6-4473-ad4c-be9e6ea72bca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "03041350-1de6-4473-ad4c-be9e6ea72bca" (UID: "03041350-1de6-4473-ad4c-be9e6ea72bca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:26:02.160751 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.160424    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03041350-1de6-4473-ad4c-be9e6ea72bca-tmp-dir\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:02.160751 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.160445    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03041350-1de6-4473-ad4c-be9e6ea72bca-model-cache\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:02.160751 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.160464    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03041350-1de6-4473-ad4c-be9e6ea72bca-home" (OuterVolumeSpecName: "home") pod "03041350-1de6-4473-ad4c-be9e6ea72bca" (UID: "03041350-1de6-4473-ad4c-be9e6ea72bca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:26:02.160751 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.160622    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6707d426-cb56-4ae4-81b1-8e9607479a43-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6707d426-cb56-4ae4-81b1-8e9607479a43" (UID: "6707d426-cb56-4ae4-81b1-8e9607479a43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:26:02.160751 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.160637    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6707d426-cb56-4ae4-81b1-8e9607479a43-home" (OuterVolumeSpecName: "home") pod "6707d426-cb56-4ae4-81b1-8e9607479a43" (UID: "6707d426-cb56-4ae4-81b1-8e9607479a43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:26:02.160751 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.160677    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1dd569-c513-4d89-95e4-57a8e96676cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "da1dd569-c513-4d89-95e4-57a8e96676cd" (UID: "da1dd569-c513-4d89-95e4-57a8e96676cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:26:02.161044 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.160923    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1dd569-c513-4d89-95e4-57a8e96676cd-home" (OuterVolumeSpecName: "home") pod "da1dd569-c513-4d89-95e4-57a8e96676cd" (UID: "da1dd569-c513-4d89-95e4-57a8e96676cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:26:02.161044 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.161024    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1dd569-c513-4d89-95e4-57a8e96676cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da1dd569-c513-4d89-95e4-57a8e96676cd" (UID: "da1dd569-c513-4d89-95e4-57a8e96676cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:26:02.161177 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.161150    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6707d426-cb56-4ae4-81b1-8e9607479a43-model-cache" (OuterVolumeSpecName: "model-cache") pod "6707d426-cb56-4ae4-81b1-8e9607479a43" (UID: "6707d426-cb56-4ae4-81b1-8e9607479a43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:26:02.162401 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.162371    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1dd569-c513-4d89-95e4-57a8e96676cd-kube-api-access-9k8jc" (OuterVolumeSpecName: "kube-api-access-9k8jc") pod "da1dd569-c513-4d89-95e4-57a8e96676cd" (UID: "da1dd569-c513-4d89-95e4-57a8e96676cd"). InnerVolumeSpecName "kube-api-access-9k8jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:26:02.162891 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.162850    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03041350-1de6-4473-ad4c-be9e6ea72bca-kube-api-access-fnnzc" (OuterVolumeSpecName: "kube-api-access-fnnzc") pod "03041350-1de6-4473-ad4c-be9e6ea72bca" (UID: "03041350-1de6-4473-ad4c-be9e6ea72bca"). InnerVolumeSpecName "kube-api-access-fnnzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:26:02.163142 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.163119    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03041350-1de6-4473-ad4c-be9e6ea72bca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "03041350-1de6-4473-ad4c-be9e6ea72bca" (UID: "03041350-1de6-4473-ad4c-be9e6ea72bca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:26:02.163518 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.163459    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6707d426-cb56-4ae4-81b1-8e9607479a43-dshm" (OuterVolumeSpecName: "dshm") pod "6707d426-cb56-4ae4-81b1-8e9607479a43" (UID: "6707d426-cb56-4ae4-81b1-8e9607479a43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:26:02.163785 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.163754    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03041350-1de6-4473-ad4c-be9e6ea72bca-dshm" (OuterVolumeSpecName: "dshm") pod "03041350-1de6-4473-ad4c-be9e6ea72bca" (UID: "03041350-1de6-4473-ad4c-be9e6ea72bca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:26:02.164270 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.164244    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da1dd569-c513-4d89-95e4-57a8e96676cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da1dd569-c513-4d89-95e4-57a8e96676cd" (UID: "da1dd569-c513-4d89-95e4-57a8e96676cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:26:02.164745 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.164722    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6707d426-cb56-4ae4-81b1-8e9607479a43-kube-api-access-hcnb4" (OuterVolumeSpecName: "kube-api-access-hcnb4") pod "6707d426-cb56-4ae4-81b1-8e9607479a43" (UID: "6707d426-cb56-4ae4-81b1-8e9607479a43"). InnerVolumeSpecName "kube-api-access-hcnb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:26:02.164871 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.164854    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6707d426-cb56-4ae4-81b1-8e9607479a43-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6707d426-cb56-4ae4-81b1-8e9607479a43" (UID: "6707d426-cb56-4ae4-81b1-8e9607479a43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:26:02.165164 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.165146    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1dd569-c513-4d89-95e4-57a8e96676cd-dshm" (OuterVolumeSpecName: "dshm") pod "da1dd569-c513-4d89-95e4-57a8e96676cd" (UID: "da1dd569-c513-4d89-95e4-57a8e96676cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:26:02.261441 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.261406    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6707d426-cb56-4ae4-81b1-8e9607479a43-tmp-dir\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:02.261441 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.261437    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da1dd569-c513-4d89-95e4-57a8e96676cd-model-cache\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:02.261441 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.261446    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03041350-1de6-4473-ad4c-be9e6ea72bca-dshm\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:02.261671 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.261454    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6707d426-cb56-4ae4-81b1-8e9607479a43-dshm\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:02.261671 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.261462    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03041350-1de6-4473-ad4c-be9e6ea72bca-home\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:02.261671 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.261469    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6707d426-cb56-4ae4-81b1-8e9607479a43-model-cache\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:02.261671 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.261477    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnnzc\" (UniqueName: \"kubernetes.io/projected/03041350-1de6-4473-ad4c-be9e6ea72bca-kube-api-access-fnnzc\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:02.261671 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.261486    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcnb4\" (UniqueName: \"kubernetes.io/projected/6707d426-cb56-4ae4-81b1-8e9607479a43-kube-api-access-hcnb4\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:02.261671 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.261508    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da1dd569-c513-4d89-95e4-57a8e96676cd-tls-certs\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:02.261671 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.261532    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da1dd569-c513-4d89-95e4-57a8e96676cd-home\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:02.261671 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.261544    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da1dd569-c513-4d89-95e4-57a8e96676cd-tmp-dir\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:02.261671 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.261554    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6707d426-cb56-4ae4-81b1-8e9607479a43-tls-certs\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:02.261671 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.261561    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da1dd569-c513-4d89-95e4-57a8e96676cd-dshm\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:02.261671 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.261569    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03041350-1de6-4473-ad4c-be9e6ea72bca-tls-certs\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:02.261671 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.261578    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k8jc\" (UniqueName: \"kubernetes.io/projected/da1dd569-c513-4d89-95e4-57a8e96676cd-kube-api-access-9k8jc\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:02.261671 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.261586    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6707d426-cb56-4ae4-81b1-8e9607479a43-home\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:02.441615 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.441582    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d36169a-7242-4916-9307-3deee76d24b1" path="/var/lib/kubelet/pods/6d36169a-7242-4916-9307-3deee76d24b1/volumes"
Sep 11 20:26:02.442013 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.441993    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3530077-5092-45e4-b1c6-33c9ee6ff373" path="/var/lib/kubelet/pods/a3530077-5092-45e4-b1c6-33c9ee6ff373/volumes"
Sep 11 20:26:02.442430 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.442412    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d943c471-9182-4add-a825-58deb30b3816" path="/var/lib/kubelet/pods/d943c471-9182-4add-a825-58deb30b3816/volumes"
Sep 11 20:26:02.794702 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.794608    2580 generic.go:356] "Generic (PLEG): container finished" podID="6707d426-cb56-4ae4-81b1-8e9607479a43" containerID="fa5e64b064aaca35405f6ed30cdbffe7ed61114e9fc2d307e5cfb3c98c01d07d" exitCode=0
Sep 11 20:26:02.794702 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.794653    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"
Sep 11 20:26:02.795143 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.794909    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs" event={"ID":"6707d426-cb56-4ae4-81b1-8e9607479a43","Type":"ContainerDied","Data":"fa5e64b064aaca35405f6ed30cdbffe7ed61114e9fc2d307e5cfb3c98c01d07d"}
Sep 11 20:26:02.795143 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.794970    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs" event={"ID":"6707d426-cb56-4ae4-81b1-8e9607479a43","Type":"ContainerDied","Data":"cda9bebc8e27e5f5053a180c31829eee55cd331b44ce0d1bc92ac147308f0b91"}
Sep 11 20:26:02.795143 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.794994    2580 scope.go:117] "RemoveContainer" containerID="fa5e64b064aaca35405f6ed30cdbffe7ed61114e9fc2d307e5cfb3c98c01d07d"
Sep 11 20:26:02.796531 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.796346    2580 generic.go:356] "Generic (PLEG): container finished" podID="03041350-1de6-4473-ad4c-be9e6ea72bca" containerID="c32758a94a5fb446c91fbc3bc457ea27a7be613b764df3c4a9b9f4fb7c8b7aca" exitCode=0
Sep 11 20:26:02.796531 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.796447    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"
Sep 11 20:26:02.796531 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.796459    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs" event={"ID":"03041350-1de6-4473-ad4c-be9e6ea72bca","Type":"ContainerDied","Data":"c32758a94a5fb446c91fbc3bc457ea27a7be613b764df3c4a9b9f4fb7c8b7aca"}
Sep 11 20:26:02.796531 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.796515    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs" event={"ID":"03041350-1de6-4473-ad4c-be9e6ea72bca","Type":"ContainerDied","Data":"fd87d7361d2b0c4324747fc8074eacb497cca7a35bd02dd88ac33ae7015cb0c1"}
Sep 11 20:26:02.798286 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.798263    2580 generic.go:356] "Generic (PLEG): container finished" podID="da1dd569-c513-4d89-95e4-57a8e96676cd" containerID="33bf36efc2bf29a05f0585ba8075f75c449209fb4a57595f656191fe639379d2" exitCode=0
Sep 11 20:26:02.798412 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.798293    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4" event={"ID":"da1dd569-c513-4d89-95e4-57a8e96676cd","Type":"ContainerDied","Data":"33bf36efc2bf29a05f0585ba8075f75c449209fb4a57595f656191fe639379d2"}
Sep 11 20:26:02.798412 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.798305    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"
Sep 11 20:26:02.798412 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.798317    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4" event={"ID":"da1dd569-c513-4d89-95e4-57a8e96676cd","Type":"ContainerDied","Data":"d429aca07597c602a3c0f52be3090157016895b515b694b91d6e84f015b5d2a5"}
Sep 11 20:26:02.812479 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.812461    2580 scope.go:117] "RemoveContainer" containerID="fa5e64b064aaca35405f6ed30cdbffe7ed61114e9fc2d307e5cfb3c98c01d07d"
Sep 11 20:26:02.812844 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:26:02.812825    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa5e64b064aaca35405f6ed30cdbffe7ed61114e9fc2d307e5cfb3c98c01d07d\": container with ID starting with fa5e64b064aaca35405f6ed30cdbffe7ed61114e9fc2d307e5cfb3c98c01d07d not found: ID does not exist" containerID="fa5e64b064aaca35405f6ed30cdbffe7ed61114e9fc2d307e5cfb3c98c01d07d"
Sep 11 20:26:02.812918 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.812849    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa5e64b064aaca35405f6ed30cdbffe7ed61114e9fc2d307e5cfb3c98c01d07d"} err="failed to get container status \"fa5e64b064aaca35405f6ed30cdbffe7ed61114e9fc2d307e5cfb3c98c01d07d\": rpc error: code = NotFound desc = could not find container \"fa5e64b064aaca35405f6ed30cdbffe7ed61114e9fc2d307e5cfb3c98c01d07d\": container with ID starting with fa5e64b064aaca35405f6ed30cdbffe7ed61114e9fc2d307e5cfb3c98c01d07d not found: ID does not exist"
Sep 11 20:26:02.812918 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.812868    2580 scope.go:117] "RemoveContainer" containerID="c32758a94a5fb446c91fbc3bc457ea27a7be613b764df3c4a9b9f4fb7c8b7aca"
Sep 11 20:26:02.821751 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.821729    2580 scope.go:117] "RemoveContainer" containerID="c32758a94a5fb446c91fbc3bc457ea27a7be613b764df3c4a9b9f4fb7c8b7aca"
Sep 11 20:26:02.822030 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:26:02.822011    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c32758a94a5fb446c91fbc3bc457ea27a7be613b764df3c4a9b9f4fb7c8b7aca\": container with ID starting with c32758a94a5fb446c91fbc3bc457ea27a7be613b764df3c4a9b9f4fb7c8b7aca not found: ID does not exist" containerID="c32758a94a5fb446c91fbc3bc457ea27a7be613b764df3c4a9b9f4fb7c8b7aca"
Sep 11 20:26:02.822100 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.822045    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c32758a94a5fb446c91fbc3bc457ea27a7be613b764df3c4a9b9f4fb7c8b7aca"} err="failed to get container status \"c32758a94a5fb446c91fbc3bc457ea27a7be613b764df3c4a9b9f4fb7c8b7aca\": rpc error: code = NotFound desc = could not find container \"c32758a94a5fb446c91fbc3bc457ea27a7be613b764df3c4a9b9f4fb7c8b7aca\": container with ID starting with c32758a94a5fb446c91fbc3bc457ea27a7be613b764df3c4a9b9f4fb7c8b7aca not found: ID does not exist"
Sep 11 20:26:02.822100 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.822070    2580 scope.go:117] "RemoveContainer" containerID="33bf36efc2bf29a05f0585ba8075f75c449209fb4a57595f656191fe639379d2"
Sep 11 20:26:02.829591 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.829566    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"]
Sep 11 20:26:02.830911 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.830897    2580 scope.go:117] "RemoveContainer" containerID="33bf36efc2bf29a05f0585ba8075f75c449209fb4a57595f656191fe639379d2"
Sep 11 20:26:02.831181 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:26:02.831162    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33bf36efc2bf29a05f0585ba8075f75c449209fb4a57595f656191fe639379d2\": container with ID starting with 33bf36efc2bf29a05f0585ba8075f75c449209fb4a57595f656191fe639379d2 not found: ID does not exist" containerID="33bf36efc2bf29a05f0585ba8075f75c449209fb4a57595f656191fe639379d2"
Sep 11 20:26:02.831247 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.831191    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33bf36efc2bf29a05f0585ba8075f75c449209fb4a57595f656191fe639379d2"} err="failed to get container status \"33bf36efc2bf29a05f0585ba8075f75c449209fb4a57595f656191fe639379d2\": rpc error: code = NotFound desc = could not find container \"33bf36efc2bf29a05f0585ba8075f75c449209fb4a57595f656191fe639379d2\": container with ID starting with 33bf36efc2bf29a05f0585ba8075f75c449209fb4a57595f656191fe639379d2 not found: ID does not exist"
Sep 11 20:26:02.835187 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.835166    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-22fc84/premium-w4-22fc84-kserve-55f7947b89-htpvs"]
Sep 11 20:26:02.851833 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.851798    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"]
Sep 11 20:26:02.855468 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.855446    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-22fc84/distinct-2-w4-22fc84-kserve-6b7fb8b47-h9tjs"]
Sep 11 20:26:02.873896 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.873862    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"]
Sep 11 20:26:02.882732 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.882705    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-22fc84/distinct-w4-22fc84-kserve-5f8c886f48-h5ws4"]
Sep 11 20:26:02.886408 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.886381    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w"]
Sep 11 20:26:02.886651 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.886626    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w" podUID="4686648d-9b0e-4d84-9a1d-eed08c0ac173" containerName="payload-processing" containerID="cri-o://89a971ef2c6575d6e97a52858860eaba75a2c626758e32c3dc2663eb975ea22b" gracePeriod=30
Sep 11 20:26:02.898741 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.898715    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8"]
Sep 11 20:26:02.898915 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:02.898896    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8" podUID="e95cdb8e-5d5c-475d-aa19-d76838021842" containerName="payload-pre-processing" containerID="cri-o://cbb0a63e00757080aa719dd02d298aa52077c5dcd845f6f7585fb93717766a12" gracePeriod=30
Sep 11 20:26:03.166209 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.166182    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w"
Sep 11 20:26:03.168884 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.168863    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8"
Sep 11 20:26:03.188224 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.188193    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3faec0-7zc88"]
Sep 11 20:26:03.204747 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.204703    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3faec0-7zc88"]
Sep 11 20:26:03.269886 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.269851    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frrgc\" (UniqueName: \"kubernetes.io/projected/e95cdb8e-5d5c-475d-aa19-d76838021842-kube-api-access-frrgc\") pod \"e95cdb8e-5d5c-475d-aa19-d76838021842\" (UID: \"e95cdb8e-5d5c-475d-aa19-d76838021842\") "
Sep 11 20:26:03.270070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.269892    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn4qn\" (UniqueName: \"kubernetes.io/projected/4686648d-9b0e-4d84-9a1d-eed08c0ac173-kube-api-access-hn4qn\") pod \"4686648d-9b0e-4d84-9a1d-eed08c0ac173\" (UID: \"4686648d-9b0e-4d84-9a1d-eed08c0ac173\") "
Sep 11 20:26:03.270070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.269922    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4686648d-9b0e-4d84-9a1d-eed08c0ac173-plugins-config-volume\") pod \"4686648d-9b0e-4d84-9a1d-eed08c0ac173\" (UID: \"4686648d-9b0e-4d84-9a1d-eed08c0ac173\") "
Sep 11 20:26:03.270070 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.269964    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95cdb8e-5d5c-475d-aa19-d76838021842-plugins-config-volume\") pod \"e95cdb8e-5d5c-475d-aa19-d76838021842\" (UID: \"e95cdb8e-5d5c-475d-aa19-d76838021842\") "
Sep 11 20:26:03.270368 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.270339    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95cdb8e-5d5c-475d-aa19-d76838021842-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e95cdb8e-5d5c-475d-aa19-d76838021842" (UID: "e95cdb8e-5d5c-475d-aa19-d76838021842"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:26:03.270368 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.270337    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4686648d-9b0e-4d84-9a1d-eed08c0ac173-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4686648d-9b0e-4d84-9a1d-eed08c0ac173" (UID: "4686648d-9b0e-4d84-9a1d-eed08c0ac173"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:26:03.271988 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.271965    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4686648d-9b0e-4d84-9a1d-eed08c0ac173-kube-api-access-hn4qn" (OuterVolumeSpecName: "kube-api-access-hn4qn") pod "4686648d-9b0e-4d84-9a1d-eed08c0ac173" (UID: "4686648d-9b0e-4d84-9a1d-eed08c0ac173"). InnerVolumeSpecName "kube-api-access-hn4qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:26:03.272098 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.272006    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e95cdb8e-5d5c-475d-aa19-d76838021842-kube-api-access-frrgc" (OuterVolumeSpecName: "kube-api-access-frrgc") pod "e95cdb8e-5d5c-475d-aa19-d76838021842" (UID: "e95cdb8e-5d5c-475d-aa19-d76838021842"). InnerVolumeSpecName "kube-api-access-frrgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:26:03.371779 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.371689    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e95cdb8e-5d5c-475d-aa19-d76838021842-plugins-config-volume\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:03.371779 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.371730    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frrgc\" (UniqueName: \"kubernetes.io/projected/e95cdb8e-5d5c-475d-aa19-d76838021842-kube-api-access-frrgc\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:03.371779 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.371740    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn4qn\" (UniqueName: \"kubernetes.io/projected/4686648d-9b0e-4d84-9a1d-eed08c0ac173-kube-api-access-hn4qn\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:03.371779 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.371749    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4686648d-9b0e-4d84-9a1d-eed08c0ac173-plugins-config-volume\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:03.806584 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.806547    2580 generic.go:356] "Generic (PLEG): container finished" podID="e95cdb8e-5d5c-475d-aa19-d76838021842" containerID="cbb0a63e00757080aa719dd02d298aa52077c5dcd845f6f7585fb93717766a12" exitCode=0
Sep 11 20:26:03.807014 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.806610    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8"
Sep 11 20:26:03.807014 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.806632    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8" event={"ID":"e95cdb8e-5d5c-475d-aa19-d76838021842","Type":"ContainerDied","Data":"cbb0a63e00757080aa719dd02d298aa52077c5dcd845f6f7585fb93717766a12"}
Sep 11 20:26:03.807014 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.806688    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8" event={"ID":"e95cdb8e-5d5c-475d-aa19-d76838021842","Type":"ContainerDied","Data":"79fea21a4b80e3bed5fe45d2c25274842637e234f99f3230ea21d2c62af3ac04"}
Sep 11 20:26:03.807014 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.806719    2580 scope.go:117] "RemoveContainer" containerID="cbb0a63e00757080aa719dd02d298aa52077c5dcd845f6f7585fb93717766a12"
Sep 11 20:26:03.809933 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.809910    2580 generic.go:356] "Generic (PLEG): container finished" podID="4686648d-9b0e-4d84-9a1d-eed08c0ac173" containerID="89a971ef2c6575d6e97a52858860eaba75a2c626758e32c3dc2663eb975ea22b" exitCode=0
Sep 11 20:26:03.810040 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.809944    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w" event={"ID":"4686648d-9b0e-4d84-9a1d-eed08c0ac173","Type":"ContainerDied","Data":"89a971ef2c6575d6e97a52858860eaba75a2c626758e32c3dc2663eb975ea22b"}
Sep 11 20:26:03.810040 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.809969    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w"
Sep 11 20:26:03.810040 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.809982    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w" event={"ID":"4686648d-9b0e-4d84-9a1d-eed08c0ac173","Type":"ContainerDied","Data":"3a3a32bcd71ab011b1937df9e79cba106ca4d893c3a03293f5b3068b1b458254"}
Sep 11 20:26:03.818189 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.818167    2580 scope.go:117] "RemoveContainer" containerID="cbb0a63e00757080aa719dd02d298aa52077c5dcd845f6f7585fb93717766a12"
Sep 11 20:26:03.818484 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:26:03.818462    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb0a63e00757080aa719dd02d298aa52077c5dcd845f6f7585fb93717766a12\": container with ID starting with cbb0a63e00757080aa719dd02d298aa52077c5dcd845f6f7585fb93717766a12 not found: ID does not exist" containerID="cbb0a63e00757080aa719dd02d298aa52077c5dcd845f6f7585fb93717766a12"
Sep 11 20:26:03.818580 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.818510    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb0a63e00757080aa719dd02d298aa52077c5dcd845f6f7585fb93717766a12"} err="failed to get container status \"cbb0a63e00757080aa719dd02d298aa52077c5dcd845f6f7585fb93717766a12\": rpc error: code = NotFound desc = could not find container \"cbb0a63e00757080aa719dd02d298aa52077c5dcd845f6f7585fb93717766a12\": container with ID starting with cbb0a63e00757080aa719dd02d298aa52077c5dcd845f6f7585fb93717766a12 not found: ID does not exist"
Sep 11 20:26:03.818580 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.818533    2580 scope.go:117] "RemoveContainer" containerID="89a971ef2c6575d6e97a52858860eaba75a2c626758e32c3dc2663eb975ea22b"
Sep 11 20:26:03.826526 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.826509    2580 scope.go:117] "RemoveContainer" containerID="89a971ef2c6575d6e97a52858860eaba75a2c626758e32c3dc2663eb975ea22b"
Sep 11 20:26:03.826789 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:26:03.826767    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a971ef2c6575d6e97a52858860eaba75a2c626758e32c3dc2663eb975ea22b\": container with ID starting with 89a971ef2c6575d6e97a52858860eaba75a2c626758e32c3dc2663eb975ea22b not found: ID does not exist" containerID="89a971ef2c6575d6e97a52858860eaba75a2c626758e32c3dc2663eb975ea22b"
Sep 11 20:26:03.826856 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.826795    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a971ef2c6575d6e97a52858860eaba75a2c626758e32c3dc2663eb975ea22b"} err="failed to get container status \"89a971ef2c6575d6e97a52858860eaba75a2c626758e32c3dc2663eb975ea22b\": rpc error: code = NotFound desc = could not find container \"89a971ef2c6575d6e97a52858860eaba75a2c626758e32c3dc2663eb975ea22b\": container with ID starting with 89a971ef2c6575d6e97a52858860eaba75a2c626758e32c3dc2663eb975ea22b not found: ID does not exist"
Sep 11 20:26:03.837886 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.837865    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8"]
Sep 11 20:26:03.860580 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.860557    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3faec0-85d77fcf58-stzq8"]
Sep 11 20:26:03.882917 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.882886    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w"]
Sep 11 20:26:03.895139 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:03.895115    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3faec0-79dc5b5bff-2rw2w"]
Sep 11 20:26:04.440424 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:04.440390    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03041350-1de6-4473-ad4c-be9e6ea72bca" path="/var/lib/kubelet/pods/03041350-1de6-4473-ad4c-be9e6ea72bca/volumes"
Sep 11 20:26:04.440789 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:04.440775    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4686648d-9b0e-4d84-9a1d-eed08c0ac173" path="/var/lib/kubelet/pods/4686648d-9b0e-4d84-9a1d-eed08c0ac173/volumes"
Sep 11 20:26:04.441062 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:04.441051    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66e3e043-ad29-497f-886c-85d232dd6aaa" path="/var/lib/kubelet/pods/66e3e043-ad29-497f-886c-85d232dd6aaa/volumes"
Sep 11 20:26:04.441696 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:04.441681    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6707d426-cb56-4ae4-81b1-8e9607479a43" path="/var/lib/kubelet/pods/6707d426-cb56-4ae4-81b1-8e9607479a43/volumes"
Sep 11 20:26:04.442057 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:04.442045    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da1dd569-c513-4d89-95e4-57a8e96676cd" path="/var/lib/kubelet/pods/da1dd569-c513-4d89-95e4-57a8e96676cd/volumes"
Sep 11 20:26:04.442356 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:04.442345    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e95cdb8e-5d5c-475d-aa19-d76838021842" path="/var/lib/kubelet/pods/e95cdb8e-5d5c-475d-aa19-d76838021842/volumes"
Sep 11 20:26:10.564125 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564084    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b0d28c-rfsdx"]
Sep 11 20:26:10.564626 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564606    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d36169a-7242-4916-9307-3deee76d24b1" containerName="payload-pre-processing"
Sep 11 20:26:10.564626 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564626    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d36169a-7242-4916-9307-3deee76d24b1" containerName="payload-pre-processing"
Sep 11 20:26:10.564756 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564640    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6707d426-cb56-4ae4-81b1-8e9607479a43" containerName="main"
Sep 11 20:26:10.564756 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564648    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6707d426-cb56-4ae4-81b1-8e9607479a43" containerName="main"
Sep 11 20:26:10.564756 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564681    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d943c471-9182-4add-a825-58deb30b3816" containerName="payload-processing"
Sep 11 20:26:10.564756 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564691    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d943c471-9182-4add-a825-58deb30b3816" containerName="payload-processing"
Sep 11 20:26:10.564756 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564700    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e95cdb8e-5d5c-475d-aa19-d76838021842" containerName="payload-pre-processing"
Sep 11 20:26:10.564756 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564708    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e95cdb8e-5d5c-475d-aa19-d76838021842" containerName="payload-pre-processing"
Sep 11 20:26:10.564756 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564719    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4686648d-9b0e-4d84-9a1d-eed08c0ac173" containerName="payload-processing"
Sep 11 20:26:10.564756 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564728    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4686648d-9b0e-4d84-9a1d-eed08c0ac173" containerName="payload-processing"
Sep 11 20:26:10.564756 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564744    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03041350-1de6-4473-ad4c-be9e6ea72bca" containerName="main"
Sep 11 20:26:10.564756 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564753    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="03041350-1de6-4473-ad4c-be9e6ea72bca" containerName="main"
Sep 11 20:26:10.565228 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564763    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da1dd569-c513-4d89-95e4-57a8e96676cd" containerName="main"
Sep 11 20:26:10.565228 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564772    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="da1dd569-c513-4d89-95e4-57a8e96676cd" containerName="main"
Sep 11 20:26:10.565228 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564785    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66e3e043-ad29-497f-886c-85d232dd6aaa" containerName="revoke-keys"
Sep 11 20:26:10.565228 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564794    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="66e3e043-ad29-497f-886c-85d232dd6aaa" containerName="revoke-keys"
Sep 11 20:26:10.565228 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564896    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d36169a-7242-4916-9307-3deee76d24b1" containerName="payload-pre-processing"
Sep 11 20:26:10.565228 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564910    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e95cdb8e-5d5c-475d-aa19-d76838021842" containerName="payload-pre-processing"
Sep 11 20:26:10.565228 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564919    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4686648d-9b0e-4d84-9a1d-eed08c0ac173" containerName="payload-processing"
Sep 11 20:26:10.565228 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564928    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="66e3e043-ad29-497f-886c-85d232dd6aaa" containerName="revoke-keys"
Sep 11 20:26:10.565228 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564938    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="da1dd569-c513-4d89-95e4-57a8e96676cd" containerName="main"
Sep 11 20:26:10.565228 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564947    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d943c471-9182-4add-a825-58deb30b3816" containerName="payload-processing"
Sep 11 20:26:10.565228 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564956    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6707d426-cb56-4ae4-81b1-8e9607479a43" containerName="main"
Sep 11 20:26:10.565228 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.564964    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="03041350-1de6-4473-ad4c-be9e6ea72bca" containerName="main"
Sep 11 20:26:10.569823 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.569803    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b0d28c-rfsdx"
Sep 11 20:26:10.573326 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.573306    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:26:10.573721 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.573701    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7dbs5\""
Sep 11 20:26:10.577461 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.577197    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b0d28c-rfsdx"]
Sep 11 20:26:10.633621 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.633588    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dd49c47-db37-4635-ba45-8dd72917c1e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b0d28c-rfsdx\" (UID: \"9dd49c47-db37-4635-ba45-8dd72917c1e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b0d28c-rfsdx"
Sep 11 20:26:10.734330 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.734303    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dd49c47-db37-4635-ba45-8dd72917c1e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b0d28c-rfsdx\" (UID: \"9dd49c47-db37-4635-ba45-8dd72917c1e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b0d28c-rfsdx"
Sep 11 20:26:10.734962 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.734940    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dd49c47-db37-4635-ba45-8dd72917c1e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b0d28c-rfsdx\" (UID: \"9dd49c47-db37-4635-ba45-8dd72917c1e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b0d28c-rfsdx"
Sep 11 20:26:10.881649 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:10.881571    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b0d28c-rfsdx"
Sep 11 20:26:11.179407 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:11.179375    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b0d28c-rfsdx"]
Sep 11 20:26:11.182049 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:26:11.182022    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd49c47_db37_4635_ba45_8dd72917c1e8.slice/crio-9f0ba3edf7b426e5c14fbe1c878188a229928bb50451c5f2dc43959095cb3d85 WatchSource:0}: Error finding container 9f0ba3edf7b426e5c14fbe1c878188a229928bb50451c5f2dc43959095cb3d85: Status 404 returned error can't find the container with id 9f0ba3edf7b426e5c14fbe1c878188a229928bb50451c5f2dc43959095cb3d85
Sep 11 20:26:11.866382 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:11.866352    2580 generic.go:356] "Generic (PLEG): container finished" podID="9dd49c47-db37-4635-ba45-8dd72917c1e8" containerID="d8a67bc3b2c48bf69eb3305c76597fa2bb0717fa7e1a04e7e6a928556abfbe0e" exitCode=0
Sep 11 20:26:11.866795 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:11.866436    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b0d28c-rfsdx" event={"ID":"9dd49c47-db37-4635-ba45-8dd72917c1e8","Type":"ContainerDied","Data":"d8a67bc3b2c48bf69eb3305c76597fa2bb0717fa7e1a04e7e6a928556abfbe0e"}
Sep 11 20:26:11.866795 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:11.866546    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b0d28c-rfsdx" event={"ID":"9dd49c47-db37-4635-ba45-8dd72917c1e8","Type":"ContainerStarted","Data":"9f0ba3edf7b426e5c14fbe1c878188a229928bb50451c5f2dc43959095cb3d85"}
Sep 11 20:26:13.004551 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:13.004525    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b0d28c-rfsdx"
Sep 11 20:26:13.157855 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:13.157773    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dd49c47-db37-4635-ba45-8dd72917c1e8-maas-api-service-ca\") pod \"9dd49c47-db37-4635-ba45-8dd72917c1e8\" (UID: \"9dd49c47-db37-4635-ba45-8dd72917c1e8\") "
Sep 11 20:26:13.158126 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:13.158104    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd49c47-db37-4635-ba45-8dd72917c1e8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9dd49c47-db37-4635-ba45-8dd72917c1e8" (UID: "9dd49c47-db37-4635-ba45-8dd72917c1e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:26:13.259109 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:13.259080    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dd49c47-db37-4635-ba45-8dd72917c1e8-maas-api-service-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:13.879426 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:13.879391    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b0d28c-rfsdx"
Sep 11 20:26:13.879620 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:13.879392    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b0d28c-rfsdx" event={"ID":"9dd49c47-db37-4635-ba45-8dd72917c1e8","Type":"ContainerDied","Data":"9f0ba3edf7b426e5c14fbe1c878188a229928bb50451c5f2dc43959095cb3d85"}
Sep 11 20:26:13.879620 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:13.879548    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f0ba3edf7b426e5c14fbe1c878188a229928bb50451c5f2dc43959095cb3d85"
Sep 11 20:26:19.413861 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:19.413825    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:26:26.187853 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:26.187821    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm"]
Sep 11 20:26:26.188301 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:26.188023    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm" podUID="9a802b7f-9c38-486e-978e-2a5188d7caf8" containerName="payload-processing" containerID="cri-o://2512ca55dcd7d66bec84db8d486fdf472edddabba57a7b28ff9fcb05ea7be866" gracePeriod=30
Sep 11 20:26:26.370566 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:26.370483    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b0d28c-rfsdx"]
Sep 11 20:26:26.374645 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:26.374613    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b0d28c-rfsdx"]
Sep 11 20:26:26.441977 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:26.441903    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dd49c47-db37-4635-ba45-8dd72917c1e8" path="/var/lib/kubelet/pods/9dd49c47-db37-4635-ba45-8dd72917c1e8/volumes"
Sep 11 20:26:26.445436 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:26.445418    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm"
Sep 11 20:26:26.573072 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:26.573035    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-998gf\" (UniqueName: \"kubernetes.io/projected/9a802b7f-9c38-486e-978e-2a5188d7caf8-kube-api-access-998gf\") pod \"9a802b7f-9c38-486e-978e-2a5188d7caf8\" (UID: \"9a802b7f-9c38-486e-978e-2a5188d7caf8\") "
Sep 11 20:26:26.573246 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:26.573107    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a802b7f-9c38-486e-978e-2a5188d7caf8-plugins-config-volume\") pod \"9a802b7f-9c38-486e-978e-2a5188d7caf8\" (UID: \"9a802b7f-9c38-486e-978e-2a5188d7caf8\") "
Sep 11 20:26:26.573532 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:26.573481    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a802b7f-9c38-486e-978e-2a5188d7caf8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9a802b7f-9c38-486e-978e-2a5188d7caf8" (UID: "9a802b7f-9c38-486e-978e-2a5188d7caf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:26:26.575270 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:26.575243    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a802b7f-9c38-486e-978e-2a5188d7caf8-kube-api-access-998gf" (OuterVolumeSpecName: "kube-api-access-998gf") pod "9a802b7f-9c38-486e-978e-2a5188d7caf8" (UID: "9a802b7f-9c38-486e-978e-2a5188d7caf8"). InnerVolumeSpecName "kube-api-access-998gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:26:26.674047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:26.674003    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-998gf\" (UniqueName: \"kubernetes.io/projected/9a802b7f-9c38-486e-978e-2a5188d7caf8-kube-api-access-998gf\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:26.674047 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:26.674040    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a802b7f-9c38-486e-978e-2a5188d7caf8-plugins-config-volume\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:26.953844 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:26.953812    2580 generic.go:356] "Generic (PLEG): container finished" podID="9a802b7f-9c38-486e-978e-2a5188d7caf8" containerID="2512ca55dcd7d66bec84db8d486fdf472edddabba57a7b28ff9fcb05ea7be866" exitCode=0
Sep 11 20:26:26.954006 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:26.953851    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm" event={"ID":"9a802b7f-9c38-486e-978e-2a5188d7caf8","Type":"ContainerDied","Data":"2512ca55dcd7d66bec84db8d486fdf472edddabba57a7b28ff9fcb05ea7be866"}
Sep 11 20:26:26.954006 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:26.953859    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm"
Sep 11 20:26:26.954006 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:26.953892    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm" event={"ID":"9a802b7f-9c38-486e-978e-2a5188d7caf8","Type":"ContainerDied","Data":"978170c085c555b7d3078d9f77bf0f04de63f84e3dcefdf049394cecea7b1420"}
Sep 11 20:26:26.954006 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:26.953914    2580 scope.go:117] "RemoveContainer" containerID="2512ca55dcd7d66bec84db8d486fdf472edddabba57a7b28ff9fcb05ea7be866"
Sep 11 20:26:26.962879 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:26.962862    2580 scope.go:117] "RemoveContainer" containerID="2512ca55dcd7d66bec84db8d486fdf472edddabba57a7b28ff9fcb05ea7be866"
Sep 11 20:26:26.963143 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:26:26.963124    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2512ca55dcd7d66bec84db8d486fdf472edddabba57a7b28ff9fcb05ea7be866\": container with ID starting with 2512ca55dcd7d66bec84db8d486fdf472edddabba57a7b28ff9fcb05ea7be866 not found: ID does not exist" containerID="2512ca55dcd7d66bec84db8d486fdf472edddabba57a7b28ff9fcb05ea7be866"
Sep 11 20:26:26.963217 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:26.963149    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2512ca55dcd7d66bec84db8d486fdf472edddabba57a7b28ff9fcb05ea7be866"} err="failed to get container status \"2512ca55dcd7d66bec84db8d486fdf472edddabba57a7b28ff9fcb05ea7be866\": rpc error: code = NotFound desc = could not find container \"2512ca55dcd7d66bec84db8d486fdf472edddabba57a7b28ff9fcb05ea7be866\": container with ID starting with 2512ca55dcd7d66bec84db8d486fdf472edddabba57a7b28ff9fcb05ea7be866 not found: ID does not exist"
Sep 11 20:26:26.980155 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:26.980129    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm"]
Sep 11 20:26:26.982233 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:26.982214    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b0d28c-6c47778fbc-m5dkm"]
Sep 11 20:26:28.440556 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:28.440527    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a802b7f-9c38-486e-978e-2a5188d7caf8" path="/var/lib/kubelet/pods/9a802b7f-9c38-486e-978e-2a5188d7caf8/volumes"
Sep 11 20:26:30.791604 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:30.791574    2580 scope.go:117] "RemoveContainer" containerID="3f3047edabf823e2713a74608eb0b17109ca3dc4749ffa773fa9d3a0b4cd7529"
Sep 11 20:26:33.747475 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:33.747437    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea327d-jqgrn"]
Sep 11 20:26:33.747982 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:33.747953    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dd49c47-db37-4635-ba45-8dd72917c1e8" containerName="revoke-keys"
Sep 11 20:26:33.747982 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:33.747973    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dd49c47-db37-4635-ba45-8dd72917c1e8" containerName="revoke-keys"
Sep 11 20:26:33.748109 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:33.747989    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a802b7f-9c38-486e-978e-2a5188d7caf8" containerName="payload-processing"
Sep 11 20:26:33.748109 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:33.747998    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a802b7f-9c38-486e-978e-2a5188d7caf8" containerName="payload-processing"
Sep 11 20:26:33.748109 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:33.748100    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a802b7f-9c38-486e-978e-2a5188d7caf8" containerName="payload-processing"
Sep 11 20:26:33.748278 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:33.748117    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dd49c47-db37-4635-ba45-8dd72917c1e8" containerName="revoke-keys"
Sep 11 20:26:33.750966 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:33.750946    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea327d-jqgrn"
Sep 11 20:26:33.753598 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:33.753573    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7dbs5\""
Sep 11 20:26:33.753695 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:33.753646    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:26:33.763754 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:33.763731    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea327d-jqgrn"]
Sep 11 20:26:33.935275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:33.935235    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f7cf772-1198-4954-9477-1493888fe4e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ea327d-jqgrn\" (UID: \"3f7cf772-1198-4954-9477-1493888fe4e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea327d-jqgrn"
Sep 11 20:26:34.036200 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:34.036117    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f7cf772-1198-4954-9477-1493888fe4e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ea327d-jqgrn\" (UID: \"3f7cf772-1198-4954-9477-1493888fe4e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea327d-jqgrn"
Sep 11 20:26:34.036777 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:34.036759    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f7cf772-1198-4954-9477-1493888fe4e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ea327d-jqgrn\" (UID: \"3f7cf772-1198-4954-9477-1493888fe4e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea327d-jqgrn"
Sep 11 20:26:34.061163 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:34.061137    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea327d-jqgrn"
Sep 11 20:26:34.348534 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:34.348509    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea327d-jqgrn"]
Sep 11 20:26:34.349911 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:26:34.349885    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7cf772_1198_4954_9477_1493888fe4e0.slice/crio-cc69cc6afdfa3326f8e7f766e40463695299e5766e27f19f88122a8c645d31cd WatchSource:0}: Error finding container cc69cc6afdfa3326f8e7f766e40463695299e5766e27f19f88122a8c645d31cd: Status 404 returned error can't find the container with id cc69cc6afdfa3326f8e7f766e40463695299e5766e27f19f88122a8c645d31cd
Sep 11 20:26:35.011744 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:35.011705    2580 generic.go:356] "Generic (PLEG): container finished" podID="3f7cf772-1198-4954-9477-1493888fe4e0" containerID="ca4dff15da2cdc7695e0321c57d21072cb22ea7285434a3271d77510bb5c52ba" exitCode=0
Sep 11 20:26:35.012158 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:35.011788    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea327d-jqgrn" event={"ID":"3f7cf772-1198-4954-9477-1493888fe4e0","Type":"ContainerDied","Data":"ca4dff15da2cdc7695e0321c57d21072cb22ea7285434a3271d77510bb5c52ba"}
Sep 11 20:26:35.012158 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:35.011864    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea327d-jqgrn" event={"ID":"3f7cf772-1198-4954-9477-1493888fe4e0","Type":"ContainerStarted","Data":"cc69cc6afdfa3326f8e7f766e40463695299e5766e27f19f88122a8c645d31cd"}
Sep 11 20:26:36.145299 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:36.145275    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea327d-jqgrn"
Sep 11 20:26:36.255920 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:36.255880    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f7cf772-1198-4954-9477-1493888fe4e0-maas-api-service-ca\") pod \"3f7cf772-1198-4954-9477-1493888fe4e0\" (UID: \"3f7cf772-1198-4954-9477-1493888fe4e0\") "
Sep 11 20:26:36.256276 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:36.256247    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f7cf772-1198-4954-9477-1493888fe4e0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f7cf772-1198-4954-9477-1493888fe4e0" (UID: "3f7cf772-1198-4954-9477-1493888fe4e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:26:36.357364 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:36.357283    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f7cf772-1198-4954-9477-1493888fe4e0-maas-api-service-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:37.024018 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:37.023991    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea327d-jqgrn"
Sep 11 20:26:37.024189 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:37.024022    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea327d-jqgrn" event={"ID":"3f7cf772-1198-4954-9477-1493888fe4e0","Type":"ContainerDied","Data":"cc69cc6afdfa3326f8e7f766e40463695299e5766e27f19f88122a8c645d31cd"}
Sep 11 20:26:37.024189 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:37.024064    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc69cc6afdfa3326f8e7f766e40463695299e5766e27f19f88122a8c645d31cd"
Sep 11 20:26:37.938719 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:37.938686    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:26:45.090637 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:45.090600    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea327d-jqgrn"]
Sep 11 20:26:45.092697 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:45.092676    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea327d-jqgrn"]
Sep 11 20:26:46.441220 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:46.441184    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f7cf772-1198-4954-9477-1493888fe4e0" path="/var/lib/kubelet/pods/3f7cf772-1198-4954-9477-1493888fe4e0/volumes"
Sep 11 20:26:48.674049 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:48.674012    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7556-1444b1"]
Sep 11 20:26:48.674420 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:48.674344    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f7cf772-1198-4954-9477-1493888fe4e0" containerName="revoke-keys"
Sep 11 20:26:48.674420 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:48.674355    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f7cf772-1198-4954-9477-1493888fe4e0" containerName="revoke-keys"
Sep 11 20:26:48.674525 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:48.674421    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f7cf772-1198-4954-9477-1493888fe4e0" containerName="revoke-keys"
Sep 11 20:26:48.678591 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:48.678574    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7556-1444b1"
Sep 11 20:26:48.681472 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:48.681449    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-94fzf\""
Sep 11 20:26:48.684744 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:48.684724    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7556-1444b1"]
Sep 11 20:26:48.739803 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:48.739767    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrn7c\" (UniqueName: \"kubernetes.io/projected/a4d094aa-d6e3-4c5b-bf33-3eab988b2018-kube-api-access-mrn7c\") pod \"test-curl-7556-1444b1\" (UID: \"a4d094aa-d6e3-4c5b-bf33-3eab988b2018\") " pod="opendatahub/test-curl-7556-1444b1"
Sep 11 20:26:48.840907 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:48.840870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrn7c\" (UniqueName: \"kubernetes.io/projected/a4d094aa-d6e3-4c5b-bf33-3eab988b2018-kube-api-access-mrn7c\") pod \"test-curl-7556-1444b1\" (UID: \"a4d094aa-d6e3-4c5b-bf33-3eab988b2018\") " pod="opendatahub/test-curl-7556-1444b1"
Sep 11 20:26:48.849877 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:48.849846    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrn7c\" (UniqueName: \"kubernetes.io/projected/a4d094aa-d6e3-4c5b-bf33-3eab988b2018-kube-api-access-mrn7c\") pod \"test-curl-7556-1444b1\" (UID: \"a4d094aa-d6e3-4c5b-bf33-3eab988b2018\") " pod="opendatahub/test-curl-7556-1444b1"
Sep 11 20:26:48.989134 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:48.989099    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7556-1444b1"
Sep 11 20:26:49.259330 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:49.259306    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7556-1444b1"]
Sep 11 20:26:49.260634 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:26:49.260606    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4d094aa_d6e3_4c5b_bf33_3eab988b2018.slice/crio-d8849a0ca2e29c4895997cac5ca27cc95c201885cc65bace66c6bfd666d296c0 WatchSource:0}: Error finding container d8849a0ca2e29c4895997cac5ca27cc95c201885cc65bace66c6bfd666d296c0: Status 404 returned error can't find the container with id d8849a0ca2e29c4895997cac5ca27cc95c201885cc65bace66c6bfd666d296c0
Sep 11 20:26:50.095869 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:50.095837    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7556-1444b1" event={"ID":"a4d094aa-d6e3-4c5b-bf33-3eab988b2018","Type":"ContainerStarted","Data":"cea274e28403ee15dbe7e0e50bf86de05a116b6cf1b5ac9f1aa5bfa96f1e4bd3"}
Sep 11 20:26:50.096257 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:50.095878    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7556-1444b1" event={"ID":"a4d094aa-d6e3-4c5b-bf33-3eab988b2018","Type":"ContainerStarted","Data":"d8849a0ca2e29c4895997cac5ca27cc95c201885cc65bace66c6bfd666d296c0"}
Sep 11 20:26:50.113644 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:50.113597    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7556-1444b1" podStartSLOduration=2.113584192 podStartE2EDuration="2.113584192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:26:50.113238493 +0000 UTC m=+1580.105400767" watchObservedRunningTime="2026-09-11 20:26:50.113584192 +0000 UTC m=+1580.105746459"
Sep 11 20:26:50.319706 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:50.319674    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:26:50.712548 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:50.712509    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7556-1444b1"]
Sep 11 20:26:50.715163 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:50.715131    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7556-1444b1"]
Sep 11 20:26:51.101545 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:26:51.101511    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7556-1444b1" container="test-curl-7556-1444b1"
Sep 11 20:26:52.107936 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:52.107894    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7556-1444b1" podUID="a4d094aa-d6e3-4c5b-bf33-3eab988b2018" containerName="test-curl-7556-1444b1" containerID="cri-o://cea274e28403ee15dbe7e0e50bf86de05a116b6cf1b5ac9f1aa5bfa96f1e4bd3" gracePeriod=2
Sep 11 20:26:52.110651 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:26:52.110620    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7556-1444b1\" is forbidden: User \"system:node:ip-10-0-128-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-209.ec2.internal' and this object" podUID="a4d094aa-d6e3-4c5b-bf33-3eab988b2018" pod="opendatahub/test-curl-7556-1444b1"
Sep 11 20:26:52.137510 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:52.134089    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"]
Sep 11 20:26:52.158765 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:26:52.158722    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7556-1444b1\" is forbidden: User \"system:node:ip-10-0-128-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-209.ec2.internal' and this object" podUID="a4d094aa-d6e3-4c5b-bf33-3eab988b2018" pod="opendatahub/test-curl-7556-1444b1"
Sep 11 20:26:52.541256 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:52.541222    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c7e744-zqst8"]
Sep 11 20:26:52.541575 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:52.541562    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4d094aa-d6e3-4c5b-bf33-3eab988b2018" containerName="test-curl-7556-1444b1"
Sep 11 20:26:52.541631 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:52.541577    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4d094aa-d6e3-4c5b-bf33-3eab988b2018" containerName="test-curl-7556-1444b1"
Sep 11 20:26:52.541678 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:52.541640    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4d094aa-d6e3-4c5b-bf33-3eab988b2018" containerName="test-curl-7556-1444b1"
Sep 11 20:26:52.544754 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:52.544738    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c7e744-zqst8"
Sep 11 20:26:52.548033 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:52.548011    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7dbs5\""
Sep 11 20:26:52.549600 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:52.549579    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:26:52.561749 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:52.561721    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c7e744-zqst8"]
Sep 11 20:26:52.669567 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:52.669535    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50289c10-5a6e-493c-8a6e-728e679bf869-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c7e744-zqst8\" (UID: \"50289c10-5a6e-493c-8a6e-728e679bf869\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c7e744-zqst8"
Sep 11 20:26:52.770326 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:52.770299    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50289c10-5a6e-493c-8a6e-728e679bf869-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c7e744-zqst8\" (UID: \"50289c10-5a6e-493c-8a6e-728e679bf869\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c7e744-zqst8"
Sep 11 20:26:52.770869 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:52.770853    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50289c10-5a6e-493c-8a6e-728e679bf869-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c7e744-zqst8\" (UID: \"50289c10-5a6e-493c-8a6e-728e679bf869\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c7e744-zqst8"
Sep 11 20:26:52.854962 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:52.854895    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c7e744-zqst8"
Sep 11 20:26:53.114508 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:53.114425    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"]
Sep 11 20:26:53.151015 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:26:53.150973    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50289c10_5a6e_493c_8a6e_728e679bf869.slice/crio-26298f937378812778b9a33f7f0ef1679c27cf2ec9f6d839db0f4d53f5ffa678 WatchSource:0}: Error finding container 26298f937378812778b9a33f7f0ef1679c27cf2ec9f6d839db0f4d53f5ffa678: Status 404 returned error can't find the container with id 26298f937378812778b9a33f7f0ef1679c27cf2ec9f6d839db0f4d53f5ffa678
Sep 11 20:26:53.151015 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:53.150980    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c7e744-zqst8"]
Sep 11 20:26:54.123627 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:54.123602    2580 generic.go:356] "Generic (PLEG): container finished" podID="50289c10-5a6e-493c-8a6e-728e679bf869" containerID="9a0b5de23c7ae27930acdc58209cd508f1812b8039eaff6377f0283ef6f87c4e" exitCode=0
Sep 11 20:26:54.123922 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:54.123686    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c7e744-zqst8" event={"ID":"50289c10-5a6e-493c-8a6e-728e679bf869","Type":"ContainerDied","Data":"9a0b5de23c7ae27930acdc58209cd508f1812b8039eaff6377f0283ef6f87c4e"}
Sep 11 20:26:54.123922 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:54.123774    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c7e744-zqst8" event={"ID":"50289c10-5a6e-493c-8a6e-728e679bf869","Type":"ContainerStarted","Data":"26298f937378812778b9a33f7f0ef1679c27cf2ec9f6d839db0f4d53f5ffa678"}
Sep 11 20:26:54.252210 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:54.252185    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7556-1444b1"
Sep 11 20:26:54.281600 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:54.281576    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrn7c\" (UniqueName: \"kubernetes.io/projected/a4d094aa-d6e3-4c5b-bf33-3eab988b2018-kube-api-access-mrn7c\") pod \"a4d094aa-d6e3-4c5b-bf33-3eab988b2018\" (UID: \"a4d094aa-d6e3-4c5b-bf33-3eab988b2018\") "
Sep 11 20:26:54.283596 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:54.283568    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d094aa-d6e3-4c5b-bf33-3eab988b2018-kube-api-access-mrn7c" (OuterVolumeSpecName: "kube-api-access-mrn7c") pod "a4d094aa-d6e3-4c5b-bf33-3eab988b2018" (UID: "a4d094aa-d6e3-4c5b-bf33-3eab988b2018"). InnerVolumeSpecName "kube-api-access-mrn7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:26:54.382290 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:54.382216    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrn7c\" (UniqueName: \"kubernetes.io/projected/a4d094aa-d6e3-4c5b-bf33-3eab988b2018-kube-api-access-mrn7c\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:54.440785 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:54.440755    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4d094aa-d6e3-4c5b-bf33-3eab988b2018" path="/var/lib/kubelet/pods/a4d094aa-d6e3-4c5b-bf33-3eab988b2018/volumes"
Sep 11 20:26:55.130385 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:55.130356    2580 generic.go:356] "Generic (PLEG): container finished" podID="a4d094aa-d6e3-4c5b-bf33-3eab988b2018" containerID="cea274e28403ee15dbe7e0e50bf86de05a116b6cf1b5ac9f1aa5bfa96f1e4bd3" exitCode=137
Sep 11 20:26:55.130828 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:55.130402    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7556-1444b1"
Sep 11 20:26:55.130828 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:55.130466    2580 scope.go:117] "RemoveContainer" containerID="cea274e28403ee15dbe7e0e50bf86de05a116b6cf1b5ac9f1aa5bfa96f1e4bd3"
Sep 11 20:26:55.140277 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:55.140259    2580 scope.go:117] "RemoveContainer" containerID="cea274e28403ee15dbe7e0e50bf86de05a116b6cf1b5ac9f1aa5bfa96f1e4bd3"
Sep 11 20:26:55.140597 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:26:55.140560    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cea274e28403ee15dbe7e0e50bf86de05a116b6cf1b5ac9f1aa5bfa96f1e4bd3\": container with ID starting with cea274e28403ee15dbe7e0e50bf86de05a116b6cf1b5ac9f1aa5bfa96f1e4bd3 not found: ID does not exist" containerID="cea274e28403ee15dbe7e0e50bf86de05a116b6cf1b5ac9f1aa5bfa96f1e4bd3"
Sep 11 20:26:55.140682 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:55.140617    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea274e28403ee15dbe7e0e50bf86de05a116b6cf1b5ac9f1aa5bfa96f1e4bd3"} err="failed to get container status \"cea274e28403ee15dbe7e0e50bf86de05a116b6cf1b5ac9f1aa5bfa96f1e4bd3\": rpc error: code = NotFound desc = could not find container \"cea274e28403ee15dbe7e0e50bf86de05a116b6cf1b5ac9f1aa5bfa96f1e4bd3\": container with ID starting with cea274e28403ee15dbe7e0e50bf86de05a116b6cf1b5ac9f1aa5bfa96f1e4bd3 not found: ID does not exist"
Sep 11 20:26:55.254851 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:55.254826    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c7e744-zqst8"
Sep 11 20:26:55.290730 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:55.290698    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50289c10-5a6e-493c-8a6e-728e679bf869-maas-api-service-ca\") pod \"50289c10-5a6e-493c-8a6e-728e679bf869\" (UID: \"50289c10-5a6e-493c-8a6e-728e679bf869\") "
Sep 11 20:26:55.291039 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:55.291013    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50289c10-5a6e-493c-8a6e-728e679bf869-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "50289c10-5a6e-493c-8a6e-728e679bf869" (UID: "50289c10-5a6e-493c-8a6e-728e679bf869"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:26:55.392414 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:55.392333    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50289c10-5a6e-493c-8a6e-728e679bf869-maas-api-service-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:26:56.138301 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:56.138271    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c7e744-zqst8"
Sep 11 20:26:56.138776 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:56.138299    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c7e744-zqst8" event={"ID":"50289c10-5a6e-493c-8a6e-728e679bf869","Type":"ContainerDied","Data":"26298f937378812778b9a33f7f0ef1679c27cf2ec9f6d839db0f4d53f5ffa678"}
Sep 11 20:26:56.138776 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:26:56.138349    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26298f937378812778b9a33f7f0ef1679c27cf2ec9f6d839db0f4d53f5ffa678"
Sep 11 20:27:02.528109 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:02.528072    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:27:03.233750 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:03.233714    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c7e744-zqst8"]
Sep 11 20:27:03.239024 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:03.238990    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c7e744-zqst8"]
Sep 11 20:27:04.440387 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:04.440351    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50289c10-5a6e-493c-8a6e-728e679bf869" path="/var/lib/kubelet/pods/50289c10-5a6e-493c-8a6e-728e679bf869/volumes"
Sep 11 20:27:07.157288 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:07.157240    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr" podUID="b7f4251b-aab5-4fb0-975b-0dd1a82f72d4" containerName="main" containerID="cri-o://656440125a180fab4f0338150690861e017d6b5a79a13b17e8d5f6f818468bac" gracePeriod=45
Sep 11 20:27:07.408283 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:07.408225    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:27:07.497683 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:07.497646    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-model-cache\") pod \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\" (UID: \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\") "
Sep 11 20:27:07.497863 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:07.497693    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-tls-certs\") pod \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\" (UID: \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\") "
Sep 11 20:27:07.497863 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:07.497715    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w757d\" (UniqueName: \"kubernetes.io/projected/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-kube-api-access-w757d\") pod \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\" (UID: \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\") "
Sep 11 20:27:07.497863 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:07.497744    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-home\") pod \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\" (UID: \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\") "
Sep 11 20:27:07.497863 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:07.497760    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-tmp-dir\") pod \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\" (UID: \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\") "
Sep 11 20:27:07.497863 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:07.497780    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-dshm\") pod \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\" (UID: \"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4\") "
Sep 11 20:27:07.498160 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:07.497951    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "b7f4251b-aab5-4fb0-975b-0dd1a82f72d4" (UID: "b7f4251b-aab5-4fb0-975b-0dd1a82f72d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:27:07.498160 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:07.498080    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b7f4251b-aab5-4fb0-975b-0dd1a82f72d4" (UID: "b7f4251b-aab5-4fb0-975b-0dd1a82f72d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:27:07.498160 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:07.498100    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-model-cache\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:27:07.498160 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:07.498100    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-home" (OuterVolumeSpecName: "home") pod "b7f4251b-aab5-4fb0-975b-0dd1a82f72d4" (UID: "b7f4251b-aab5-4fb0-975b-0dd1a82f72d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:27:07.500020 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:07.499993    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-dshm" (OuterVolumeSpecName: "dshm") pod "b7f4251b-aab5-4fb0-975b-0dd1a82f72d4" (UID: "b7f4251b-aab5-4fb0-975b-0dd1a82f72d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:27:07.500020 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:07.499999    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b7f4251b-aab5-4fb0-975b-0dd1a82f72d4" (UID: "b7f4251b-aab5-4fb0-975b-0dd1a82f72d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:27:07.500170 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:07.500016    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-kube-api-access-w757d" (OuterVolumeSpecName: "kube-api-access-w757d") pod "b7f4251b-aab5-4fb0-975b-0dd1a82f72d4" (UID: "b7f4251b-aab5-4fb0-975b-0dd1a82f72d4"). InnerVolumeSpecName "kube-api-access-w757d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:27:07.598749 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:07.598720    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-tls-certs\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:27:07.598836 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:07.598749    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w757d\" (UniqueName: \"kubernetes.io/projected/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-kube-api-access-w757d\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:27:07.598836 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:07.598763    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-home\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:27:07.598836 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:07.598792    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-tmp-dir\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:27:07.598836 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:07.598801    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4-dshm\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:27:08.132696 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.132658    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk" podUID="4ccbaa69-29e9-462f-9687-98a85b69ed26" containerName="main" containerID="cri-o://f557b698f9e4a4af77a625fb959fd78c504ee2ad055246d92aeac1206abe8e5e" gracePeriod=45
Sep 11 20:27:08.205640 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.205614    2580 generic.go:356] "Generic (PLEG): container finished" podID="b7f4251b-aab5-4fb0-975b-0dd1a82f72d4" containerID="656440125a180fab4f0338150690861e017d6b5a79a13b17e8d5f6f818468bac" exitCode=0
Sep 11 20:27:08.205947 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.205660    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"
Sep 11 20:27:08.205947 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.205692    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr" event={"ID":"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4","Type":"ContainerDied","Data":"656440125a180fab4f0338150690861e017d6b5a79a13b17e8d5f6f818468bac"}
Sep 11 20:27:08.205947 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.205747    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr" event={"ID":"b7f4251b-aab5-4fb0-975b-0dd1a82f72d4","Type":"ContainerDied","Data":"877f5e7c8422e0b0ed12d7b1cb1ab699d44cc2adb70b6fa351dc57f410026a65"}
Sep 11 20:27:08.205947 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.205770    2580 scope.go:117] "RemoveContainer" containerID="656440125a180fab4f0338150690861e017d6b5a79a13b17e8d5f6f818468bac"
Sep 11 20:27:08.228065 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.228042    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"]
Sep 11 20:27:08.235527 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.235507    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcb0e4/auth-test-model-fcb0e4-kserve-6db44b99dc-vbrdr"]
Sep 11 20:27:08.244057 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.244028    2580 scope.go:117] "RemoveContainer" containerID="656440125a180fab4f0338150690861e017d6b5a79a13b17e8d5f6f818468bac"
Sep 11 20:27:08.244393 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:27:08.244374    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"656440125a180fab4f0338150690861e017d6b5a79a13b17e8d5f6f818468bac\": container with ID starting with 656440125a180fab4f0338150690861e017d6b5a79a13b17e8d5f6f818468bac not found: ID does not exist" containerID="656440125a180fab4f0338150690861e017d6b5a79a13b17e8d5f6f818468bac"
Sep 11 20:27:08.244464 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.244401    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"656440125a180fab4f0338150690861e017d6b5a79a13b17e8d5f6f818468bac"} err="failed to get container status \"656440125a180fab4f0338150690861e017d6b5a79a13b17e8d5f6f818468bac\": rpc error: code = NotFound desc = could not find container \"656440125a180fab4f0338150690861e017d6b5a79a13b17e8d5f6f818468bac\": container with ID starting with 656440125a180fab4f0338150690861e017d6b5a79a13b17e8d5f6f818468bac not found: ID does not exist"
Sep 11 20:27:08.381131 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.381106    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:27:08.440885 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.440848    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7f4251b-aab5-4fb0-975b-0dd1a82f72d4" path="/var/lib/kubelet/pods/b7f4251b-aab5-4fb0-975b-0dd1a82f72d4/volumes"
Sep 11 20:27:08.505681 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.505640    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccbaa69-29e9-462f-9687-98a85b69ed26-model-cache\") pod \"4ccbaa69-29e9-462f-9687-98a85b69ed26\" (UID: \"4ccbaa69-29e9-462f-9687-98a85b69ed26\") "
Sep 11 20:27:08.505860 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.505722    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccbaa69-29e9-462f-9687-98a85b69ed26-dshm\") pod \"4ccbaa69-29e9-462f-9687-98a85b69ed26\" (UID: \"4ccbaa69-29e9-462f-9687-98a85b69ed26\") "
Sep 11 20:27:08.505860 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.505742    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccbaa69-29e9-462f-9687-98a85b69ed26-tmp-dir\") pod \"4ccbaa69-29e9-462f-9687-98a85b69ed26\" (UID: \"4ccbaa69-29e9-462f-9687-98a85b69ed26\") "
Sep 11 20:27:08.505860 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.505769    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ccbaa69-29e9-462f-9687-98a85b69ed26-home\") pod \"4ccbaa69-29e9-462f-9687-98a85b69ed26\" (UID: \"4ccbaa69-29e9-462f-9687-98a85b69ed26\") "
Sep 11 20:27:08.505860 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.505787    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8zqf\" (UniqueName: \"kubernetes.io/projected/4ccbaa69-29e9-462f-9687-98a85b69ed26-kube-api-access-k8zqf\") pod \"4ccbaa69-29e9-462f-9687-98a85b69ed26\" (UID: \"4ccbaa69-29e9-462f-9687-98a85b69ed26\") "
Sep 11 20:27:08.505860 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.505854    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccbaa69-29e9-462f-9687-98a85b69ed26-tls-certs\") pod \"4ccbaa69-29e9-462f-9687-98a85b69ed26\" (UID: \"4ccbaa69-29e9-462f-9687-98a85b69ed26\") "
Sep 11 20:27:08.506113 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.505902    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccbaa69-29e9-462f-9687-98a85b69ed26-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ccbaa69-29e9-462f-9687-98a85b69ed26" (UID: "4ccbaa69-29e9-462f-9687-98a85b69ed26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:27:08.506113 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.506018    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccbaa69-29e9-462f-9687-98a85b69ed26-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ccbaa69-29e9-462f-9687-98a85b69ed26" (UID: "4ccbaa69-29e9-462f-9687-98a85b69ed26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:27:08.506113 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.506047    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccbaa69-29e9-462f-9687-98a85b69ed26-home" (OuterVolumeSpecName: "home") pod "4ccbaa69-29e9-462f-9687-98a85b69ed26" (UID: "4ccbaa69-29e9-462f-9687-98a85b69ed26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:27:08.506113 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.506097    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ccbaa69-29e9-462f-9687-98a85b69ed26-tmp-dir\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:27:08.506113 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.506110    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ccbaa69-29e9-462f-9687-98a85b69ed26-model-cache\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:27:08.508051 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.508023    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ccbaa69-29e9-462f-9687-98a85b69ed26-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ccbaa69-29e9-462f-9687-98a85b69ed26" (UID: "4ccbaa69-29e9-462f-9687-98a85b69ed26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:27:08.508171 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.508071    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccbaa69-29e9-462f-9687-98a85b69ed26-kube-api-access-k8zqf" (OuterVolumeSpecName: "kube-api-access-k8zqf") pod "4ccbaa69-29e9-462f-9687-98a85b69ed26" (UID: "4ccbaa69-29e9-462f-9687-98a85b69ed26"). InnerVolumeSpecName "kube-api-access-k8zqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:27:08.508171 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.508095    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccbaa69-29e9-462f-9687-98a85b69ed26-dshm" (OuterVolumeSpecName: "dshm") pod "4ccbaa69-29e9-462f-9687-98a85b69ed26" (UID: "4ccbaa69-29e9-462f-9687-98a85b69ed26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:27:08.606925 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.606897    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ccbaa69-29e9-462f-9687-98a85b69ed26-dshm\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:27:08.606925 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.606919    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ccbaa69-29e9-462f-9687-98a85b69ed26-home\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:27:08.606925 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.606929    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8zqf\" (UniqueName: \"kubernetes.io/projected/4ccbaa69-29e9-462f-9687-98a85b69ed26-kube-api-access-k8zqf\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:27:08.607127 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:08.606940    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ccbaa69-29e9-462f-9687-98a85b69ed26-tls-certs\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:27:09.030735 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.030702    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"]
Sep 11 20:27:09.031071 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.031058    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50289c10-5a6e-493c-8a6e-728e679bf869" containerName="revoke-keys"
Sep 11 20:27:09.031112 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.031073    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="50289c10-5a6e-493c-8a6e-728e679bf869" containerName="revoke-keys"
Sep 11 20:27:09.031112 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.031094    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ccbaa69-29e9-462f-9687-98a85b69ed26" containerName="main"
Sep 11 20:27:09.031112 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.031100    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ccbaa69-29e9-462f-9687-98a85b69ed26" containerName="main"
Sep 11 20:27:09.031207 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.031116    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7f4251b-aab5-4fb0-975b-0dd1a82f72d4" containerName="main"
Sep 11 20:27:09.031207 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.031124    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7f4251b-aab5-4fb0-975b-0dd1a82f72d4" containerName="main"
Sep 11 20:27:09.031207 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.031178    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ccbaa69-29e9-462f-9687-98a85b69ed26" containerName="main"
Sep 11 20:27:09.031207 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.031188    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7f4251b-aab5-4fb0-975b-0dd1a82f72d4" containerName="main"
Sep 11 20:27:09.031207 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.031200    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="50289c10-5a6e-493c-8a6e-728e679bf869" containerName="revoke-keys"
Sep 11 20:27:09.034417 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.034400    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:27:09.038683 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.038664    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-fcb0e4\"/\"kube-root-ca.crt\""
Sep 11 20:27:09.039765 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.039734    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-fcb0e4\"/\"openshift-service-ca.crt\""
Sep 11 20:27:09.039765 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.039750    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-fcb0e4\"/\"sub-test-model-fcb0e4-kserve-self-signed-certs\""
Sep 11 20:27:09.039919 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.039797    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-fcb0e4\"/\"default-dockercfg-95dbk\""
Sep 11 20:27:09.049695 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.049672    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"]
Sep 11 20:27:09.111939 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.111906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fc4658-e8e1-4242-b593-2fdae0e8781a-tls-certs\") pod \"sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv\" (UID: \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:27:09.112077 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.111944    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fc4658-e8e1-4242-b593-2fdae0e8781a-dshm\") pod \"sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv\" (UID: \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:27:09.112077 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.111986    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fc4658-e8e1-4242-b593-2fdae0e8781a-tmp-dir\") pod \"sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv\" (UID: \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:27:09.112077 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.112017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fc4658-e8e1-4242-b593-2fdae0e8781a-model-cache\") pod \"sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv\" (UID: \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:27:09.112077 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.112042    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n4w8\" (UniqueName: \"kubernetes.io/projected/a5fc4658-e8e1-4242-b593-2fdae0e8781a-kube-api-access-8n4w8\") pod \"sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv\" (UID: \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:27:09.112235 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.112080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fc4658-e8e1-4242-b593-2fdae0e8781a-home\") pod \"sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv\" (UID: \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:27:09.212506 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.212457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n4w8\" (UniqueName: \"kubernetes.io/projected/a5fc4658-e8e1-4242-b593-2fdae0e8781a-kube-api-access-8n4w8\") pod \"sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv\" (UID: \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:27:09.212955 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.212537    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fc4658-e8e1-4242-b593-2fdae0e8781a-home\") pod \"sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv\" (UID: \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:27:09.212955 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.212592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fc4658-e8e1-4242-b593-2fdae0e8781a-tls-certs\") pod \"sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv\" (UID: \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:27:09.212955 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.212607    2580 generic.go:356] "Generic (PLEG): container finished" podID="4ccbaa69-29e9-462f-9687-98a85b69ed26" containerID="f557b698f9e4a4af77a625fb959fd78c504ee2ad055246d92aeac1206abe8e5e" exitCode=0
Sep 11 20:27:09.212955 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.212626    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fc4658-e8e1-4242-b593-2fdae0e8781a-dshm\") pod \"sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv\" (UID: \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:27:09.212955 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.212668    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fc4658-e8e1-4242-b593-2fdae0e8781a-tmp-dir\") pod \"sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv\" (UID: \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:27:09.212955 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.212728    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"
Sep 11 20:27:09.212955 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.212743    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk" event={"ID":"4ccbaa69-29e9-462f-9687-98a85b69ed26","Type":"ContainerDied","Data":"f557b698f9e4a4af77a625fb959fd78c504ee2ad055246d92aeac1206abe8e5e"}
Sep 11 20:27:09.212955 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.212807    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk" event={"ID":"4ccbaa69-29e9-462f-9687-98a85b69ed26","Type":"ContainerDied","Data":"d27303a847a97736831eeec50d5da5556857189b4aba7c0d3f753391fcf91f6c"}
Sep 11 20:27:09.212955 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.212829    2580 scope.go:117] "RemoveContainer" containerID="f557b698f9e4a4af77a625fb959fd78c504ee2ad055246d92aeac1206abe8e5e"
Sep 11 20:27:09.212955 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.212726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fc4658-e8e1-4242-b593-2fdae0e8781a-model-cache\") pod \"sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv\" (UID: \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:27:09.213461 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.212970    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fc4658-e8e1-4242-b593-2fdae0e8781a-home\") pod \"sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv\" (UID: \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:27:09.213461 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.213021    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fc4658-e8e1-4242-b593-2fdae0e8781a-model-cache\") pod \"sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv\" (UID: \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:27:09.213461 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.213060    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fc4658-e8e1-4242-b593-2fdae0e8781a-tmp-dir\") pod \"sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv\" (UID: \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:27:09.215322 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.215300    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fc4658-e8e1-4242-b593-2fdae0e8781a-dshm\") pod \"sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv\" (UID: \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:27:09.215573 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.215553    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fc4658-e8e1-4242-b593-2fdae0e8781a-tls-certs\") pod \"sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv\" (UID: \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:27:09.221319 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.221301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n4w8\" (UniqueName: \"kubernetes.io/projected/a5fc4658-e8e1-4242-b593-2fdae0e8781a-kube-api-access-8n4w8\") pod \"sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv\" (UID: \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:27:09.234921 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.234903    2580 scope.go:117] "RemoveContainer" containerID="f557b698f9e4a4af77a625fb959fd78c504ee2ad055246d92aeac1206abe8e5e"
Sep 11 20:27:09.235221 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:27:09.235201    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f557b698f9e4a4af77a625fb959fd78c504ee2ad055246d92aeac1206abe8e5e\": container with ID starting with f557b698f9e4a4af77a625fb959fd78c504ee2ad055246d92aeac1206abe8e5e not found: ID does not exist" containerID="f557b698f9e4a4af77a625fb959fd78c504ee2ad055246d92aeac1206abe8e5e"
Sep 11 20:27:09.235272 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.235229    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f557b698f9e4a4af77a625fb959fd78c504ee2ad055246d92aeac1206abe8e5e"} err="failed to get container status \"f557b698f9e4a4af77a625fb959fd78c504ee2ad055246d92aeac1206abe8e5e\": rpc error: code = NotFound desc = could not find container \"f557b698f9e4a4af77a625fb959fd78c504ee2ad055246d92aeac1206abe8e5e\": container with ID starting with f557b698f9e4a4af77a625fb959fd78c504ee2ad055246d92aeac1206abe8e5e not found: ID does not exist"
Sep 11 20:27:09.256551 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.256525    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"]
Sep 11 20:27:09.259273 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.259252    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-092e1a/auth-test-model-092e1a-kserve-6d74ddff75-xprqk"]
Sep 11 20:27:09.344390 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.344333    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:27:09.622959 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:09.622936    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"]
Sep 11 20:27:09.624239 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:27:09.624207    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5fc4658_e8e1_4242_b593_2fdae0e8781a.slice/crio-b9973f985dcb4a97d675ffd636df11992402b18f4b850f3d5fb142fdf5986318 WatchSource:0}: Error finding container b9973f985dcb4a97d675ffd636df11992402b18f4b850f3d5fb142fdf5986318: Status 404 returned error can't find the container with id b9973f985dcb4a97d675ffd636df11992402b18f4b850f3d5fb142fdf5986318
Sep 11 20:27:10.222561 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:10.222520    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv" event={"ID":"a5fc4658-e8e1-4242-b593-2fdae0e8781a","Type":"ContainerStarted","Data":"2b51b7945f40a57740f84e9b679bf973173c3b38aae552252c4f0499ce594177"}
Sep 11 20:27:10.223013 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:10.222574    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv" event={"ID":"a5fc4658-e8e1-4242-b593-2fdae0e8781a","Type":"ContainerStarted","Data":"b9973f985dcb4a97d675ffd636df11992402b18f4b850f3d5fb142fdf5986318"}
Sep 11 20:27:10.247435 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:10.247384    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv" podStartSLOduration=1.247368756 podStartE2EDuration="1.247368756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:27:10.246125746 +0000 UTC m=+1600.238288017" watchObservedRunningTime="2026-09-11 20:27:10.247368756 +0000 UTC m=+1600.239531025"
Sep 11 20:27:10.441458 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:10.441425    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ccbaa69-29e9-462f-9687-98a85b69ed26" path="/var/lib/kubelet/pods/4ccbaa69-29e9-462f-9687-98a85b69ed26/volumes"
Sep 11 20:27:11.821637 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:11.821602    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:27:19.345284 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:19.345231    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:27:19.345818 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:19.345315    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:27:19.358076 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:19.358052    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:27:20.288712 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:20.288683    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:27:30.847685 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:30.847654    2580 scope.go:117] "RemoveContainer" containerID="855545f15ab65fc8fa0ed2842278cf50e7c01ea3779e11150d77a073bf52c361"
Sep 11 20:27:30.856955 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:30.856935    2580 scope.go:117] "RemoveContainer" containerID="f7732a66540f9f03a8d012d0de010c90ea012cc2841f60b481d4333cd726125c"
Sep 11 20:27:34.103356 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.103322    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"]
Sep 11 20:27:34.106739 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.106719    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:27:34.109801 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.109775    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-092e1a\"/\"openshift-service-ca.crt\""
Sep 11 20:27:34.110785 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.110767    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-092e1a\"/\"default-dockercfg-2slpb\""
Sep 11 20:27:34.110879 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.110800    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-092e1a\"/\"kube-root-ca.crt\""
Sep 11 20:27:34.111088 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.111073    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-092e1a\"/\"sub-test-model-092e1a-kserve-self-signed-certs\""
Sep 11 20:27:34.117903 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.117881    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"]
Sep 11 20:27:34.228401 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.228366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97rl2\" (UniqueName: \"kubernetes.io/projected/d2df8979-05ee-4a91-a18b-fb448eb789f1-kube-api-access-97rl2\") pod \"sub-test-model-092e1a-kserve-769b64d88c-hpsnp\" (UID: \"d2df8979-05ee-4a91-a18b-fb448eb789f1\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:27:34.228591 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.228429    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2df8979-05ee-4a91-a18b-fb448eb789f1-tmp-dir\") pod \"sub-test-model-092e1a-kserve-769b64d88c-hpsnp\" (UID: \"d2df8979-05ee-4a91-a18b-fb448eb789f1\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:27:34.228591 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.228460    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2df8979-05ee-4a91-a18b-fb448eb789f1-model-cache\") pod \"sub-test-model-092e1a-kserve-769b64d88c-hpsnp\" (UID: \"d2df8979-05ee-4a91-a18b-fb448eb789f1\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:27:34.228591 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.228475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2df8979-05ee-4a91-a18b-fb448eb789f1-tls-certs\") pod \"sub-test-model-092e1a-kserve-769b64d88c-hpsnp\" (UID: \"d2df8979-05ee-4a91-a18b-fb448eb789f1\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:27:34.228591 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.228552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2df8979-05ee-4a91-a18b-fb448eb789f1-home\") pod \"sub-test-model-092e1a-kserve-769b64d88c-hpsnp\" (UID: \"d2df8979-05ee-4a91-a18b-fb448eb789f1\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:27:34.228739 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.228592    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2df8979-05ee-4a91-a18b-fb448eb789f1-dshm\") pod \"sub-test-model-092e1a-kserve-769b64d88c-hpsnp\" (UID: \"d2df8979-05ee-4a91-a18b-fb448eb789f1\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:27:34.332516 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.330065    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2df8979-05ee-4a91-a18b-fb448eb789f1-home\") pod \"sub-test-model-092e1a-kserve-769b64d88c-hpsnp\" (UID: \"d2df8979-05ee-4a91-a18b-fb448eb789f1\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:27:34.332516 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.330156    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2df8979-05ee-4a91-a18b-fb448eb789f1-dshm\") pod \"sub-test-model-092e1a-kserve-769b64d88c-hpsnp\" (UID: \"d2df8979-05ee-4a91-a18b-fb448eb789f1\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:27:34.332516 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.330198    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97rl2\" (UniqueName: \"kubernetes.io/projected/d2df8979-05ee-4a91-a18b-fb448eb789f1-kube-api-access-97rl2\") pod \"sub-test-model-092e1a-kserve-769b64d88c-hpsnp\" (UID: \"d2df8979-05ee-4a91-a18b-fb448eb789f1\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:27:34.332516 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.330265    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2df8979-05ee-4a91-a18b-fb448eb789f1-tmp-dir\") pod \"sub-test-model-092e1a-kserve-769b64d88c-hpsnp\" (UID: \"d2df8979-05ee-4a91-a18b-fb448eb789f1\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:27:34.332516 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.330305    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2df8979-05ee-4a91-a18b-fb448eb789f1-model-cache\") pod \"sub-test-model-092e1a-kserve-769b64d88c-hpsnp\" (UID: \"d2df8979-05ee-4a91-a18b-fb448eb789f1\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:27:34.332516 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.330332    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2df8979-05ee-4a91-a18b-fb448eb789f1-tls-certs\") pod \"sub-test-model-092e1a-kserve-769b64d88c-hpsnp\" (UID: \"d2df8979-05ee-4a91-a18b-fb448eb789f1\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:27:34.334525 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.333903    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2df8979-05ee-4a91-a18b-fb448eb789f1-tls-certs\") pod \"sub-test-model-092e1a-kserve-769b64d88c-hpsnp\" (UID: \"d2df8979-05ee-4a91-a18b-fb448eb789f1\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:27:34.334525 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.334222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2df8979-05ee-4a91-a18b-fb448eb789f1-home\") pod \"sub-test-model-092e1a-kserve-769b64d88c-hpsnp\" (UID: \"d2df8979-05ee-4a91-a18b-fb448eb789f1\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:27:34.336324 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.336300    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2df8979-05ee-4a91-a18b-fb448eb789f1-tmp-dir\") pod \"sub-test-model-092e1a-kserve-769b64d88c-hpsnp\" (UID: \"d2df8979-05ee-4a91-a18b-fb448eb789f1\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:27:34.336609 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.336578    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2df8979-05ee-4a91-a18b-fb448eb789f1-model-cache\") pod \"sub-test-model-092e1a-kserve-769b64d88c-hpsnp\" (UID: \"d2df8979-05ee-4a91-a18b-fb448eb789f1\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:27:34.336805 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.336784    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2df8979-05ee-4a91-a18b-fb448eb789f1-dshm\") pod \"sub-test-model-092e1a-kserve-769b64d88c-hpsnp\" (UID: \"d2df8979-05ee-4a91-a18b-fb448eb789f1\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:27:34.348594 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.348572    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97rl2\" (UniqueName: \"kubernetes.io/projected/d2df8979-05ee-4a91-a18b-fb448eb789f1-kube-api-access-97rl2\") pod \"sub-test-model-092e1a-kserve-769b64d88c-hpsnp\" (UID: \"d2df8979-05ee-4a91-a18b-fb448eb789f1\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:27:34.416537 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.416445    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:27:34.694444 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:34.694414    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"]
Sep 11 20:27:34.696288 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:27:34.696253    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2df8979_05ee_4a91_a18b_fb448eb789f1.slice/crio-480b52e4ee9dc44fafdd6b9e67ef84ff8f9b65dd1cba95cf53071774994442a0 WatchSource:0}: Error finding container 480b52e4ee9dc44fafdd6b9e67ef84ff8f9b65dd1cba95cf53071774994442a0: Status 404 returned error can't find the container with id 480b52e4ee9dc44fafdd6b9e67ef84ff8f9b65dd1cba95cf53071774994442a0
Sep 11 20:27:35.359007 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:35.358974    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp" event={"ID":"d2df8979-05ee-4a91-a18b-fb448eb789f1","Type":"ContainerStarted","Data":"e695d885ff22a8d4e1bcc3cdb7a0e258259ce42401e22081463491a2839ea2f3"}
Sep 11 20:27:35.359514 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:35.359019    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp" event={"ID":"d2df8979-05ee-4a91-a18b-fb448eb789f1","Type":"ContainerStarted","Data":"480b52e4ee9dc44fafdd6b9e67ef84ff8f9b65dd1cba95cf53071774994442a0"}
Sep 11 20:27:35.380194 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:35.380143    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp" podStartSLOduration=1.3801280839999999 podStartE2EDuration="1.380128084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:27:35.377447697 +0000 UTC m=+1625.369609967" watchObservedRunningTime="2026-09-11 20:27:35.380128084 +0000 UTC m=+1625.372290353"
Sep 11 20:27:38.128694 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:38.128661    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:27:44.417402 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:44.417365    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:27:44.417863 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:44.417415    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:27:44.430055 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:44.430030    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:27:45.426738 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:45.426702    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:27:53.519894 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:27:53.519862    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:28:06.522674 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:06.522643    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:28:08.804713 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:08.804681    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7556-d16180"]
Sep 11 20:28:08.808131 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:08.808113    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7556-d16180"
Sep 11 20:28:08.811145 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:08.811130    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-94fzf\""
Sep 11 20:28:08.815377 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:08.815349    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7556-d16180"]
Sep 11 20:28:08.929607 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:08.929557    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68p87\" (UniqueName: \"kubernetes.io/projected/0ec990cd-5a56-4a82-aa40-08ad37696ae8-kube-api-access-68p87\") pod \"test-curl-7556-d16180\" (UID: \"0ec990cd-5a56-4a82-aa40-08ad37696ae8\") " pod="opendatahub/test-curl-7556-d16180"
Sep 11 20:28:09.030601 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:09.030564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68p87\" (UniqueName: \"kubernetes.io/projected/0ec990cd-5a56-4a82-aa40-08ad37696ae8-kube-api-access-68p87\") pod \"test-curl-7556-d16180\" (UID: \"0ec990cd-5a56-4a82-aa40-08ad37696ae8\") " pod="opendatahub/test-curl-7556-d16180"
Sep 11 20:28:09.039872 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:09.039849    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68p87\" (UniqueName: \"kubernetes.io/projected/0ec990cd-5a56-4a82-aa40-08ad37696ae8-kube-api-access-68p87\") pod \"test-curl-7556-d16180\" (UID: \"0ec990cd-5a56-4a82-aa40-08ad37696ae8\") " pod="opendatahub/test-curl-7556-d16180"
Sep 11 20:28:09.118884 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:09.118800    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7556-d16180"
Sep 11 20:28:09.391320 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:09.391295    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7556-d16180"]
Sep 11 20:28:09.393397 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:28:09.393364    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ec990cd_5a56_4a82_aa40_08ad37696ae8.slice/crio-08b891fca4da0a98d939074bbb3157448769bfff544390156986bb1670c720d3 WatchSource:0}: Error finding container 08b891fca4da0a98d939074bbb3157448769bfff544390156986bb1670c720d3: Status 404 returned error can't find the container with id 08b891fca4da0a98d939074bbb3157448769bfff544390156986bb1670c720d3
Sep 11 20:28:09.545516 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:09.545465    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7556-d16180" event={"ID":"0ec990cd-5a56-4a82-aa40-08ad37696ae8","Type":"ContainerStarted","Data":"befad8e91c9d2e16b4a9893dff49a3766d275afb6cbca811e7bb9a99deaaca8d"}
Sep 11 20:28:09.545516 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:09.545518    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7556-d16180" event={"ID":"0ec990cd-5a56-4a82-aa40-08ad37696ae8","Type":"ContainerStarted","Data":"08b891fca4da0a98d939074bbb3157448769bfff544390156986bb1670c720d3"}
Sep 11 20:28:09.566692 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:09.566644    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7556-d16180" podStartSLOduration=1.566629345 podStartE2EDuration="1.566629345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:28:09.563157484 +0000 UTC m=+1659.555319754" watchObservedRunningTime="2026-09-11 20:28:09.566629345 +0000 UTC m=+1659.558791665"
Sep 11 20:28:10.150556 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:10.150454    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7556-d16180"]
Sep 11 20:28:10.155687 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:10.155663    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7556-d16180"]
Sep 11 20:28:10.269092 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:10.269053    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7556-545f33"]
Sep 11 20:28:10.269435 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:10.269422    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ec990cd-5a56-4a82-aa40-08ad37696ae8" containerName="test-curl-7556-d16180"
Sep 11 20:28:10.269480 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:10.269436    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ec990cd-5a56-4a82-aa40-08ad37696ae8" containerName="test-curl-7556-d16180"
Sep 11 20:28:10.269534 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:10.269519    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ec990cd-5a56-4a82-aa40-08ad37696ae8" containerName="test-curl-7556-d16180"
Sep 11 20:28:10.272569 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:10.272552    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7556-545f33"
Sep 11 20:28:10.284469 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:10.284430    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7556-545f33"]
Sep 11 20:28:10.342881 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:10.342849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmhnm\" (UniqueName: \"kubernetes.io/projected/927e8641-5ec7-4cf4-abc6-99450a4e02b6-kube-api-access-dmhnm\") pod \"test-curl-7556-545f33\" (UID: \"927e8641-5ec7-4cf4-abc6-99450a4e02b6\") " pod="opendatahub/test-curl-7556-545f33"
Sep 11 20:28:10.444044 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:10.444014    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmhnm\" (UniqueName: \"kubernetes.io/projected/927e8641-5ec7-4cf4-abc6-99450a4e02b6-kube-api-access-dmhnm\") pod \"test-curl-7556-545f33\" (UID: \"927e8641-5ec7-4cf4-abc6-99450a4e02b6\") " pod="opendatahub/test-curl-7556-545f33"
Sep 11 20:28:10.453305 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:10.453279    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmhnm\" (UniqueName: \"kubernetes.io/projected/927e8641-5ec7-4cf4-abc6-99450a4e02b6-kube-api-access-dmhnm\") pod \"test-curl-7556-545f33\" (UID: \"927e8641-5ec7-4cf4-abc6-99450a4e02b6\") " pod="opendatahub/test-curl-7556-545f33"
Sep 11 20:28:10.550292 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:28:10.550266    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7556-d16180" container="test-curl-7556-d16180"
Sep 11 20:28:10.584336 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:10.584310    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7556-545f33"
Sep 11 20:28:10.862305 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:10.862280    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7556-545f33"]
Sep 11 20:28:10.863793 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:28:10.863766    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod927e8641_5ec7_4cf4_abc6_99450a4e02b6.slice/crio-f31b3d07cf23f3deb8373ca5451f761559e6136a7daf75fe299801962b18272c WatchSource:0}: Error finding container f31b3d07cf23f3deb8373ca5451f761559e6136a7daf75fe299801962b18272c: Status 404 returned error can't find the container with id f31b3d07cf23f3deb8373ca5451f761559e6136a7daf75fe299801962b18272c
Sep 11 20:28:11.557163 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:11.557127    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7556-545f33" event={"ID":"927e8641-5ec7-4cf4-abc6-99450a4e02b6","Type":"ContainerStarted","Data":"d55115c7468828213f5d66ab93d68e9ee07a30e3ede8e87e8e1215cc526312dc"}
Sep 11 20:28:11.557576 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:11.557173    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7556-545f33" event={"ID":"927e8641-5ec7-4cf4-abc6-99450a4e02b6","Type":"ContainerStarted","Data":"f31b3d07cf23f3deb8373ca5451f761559e6136a7daf75fe299801962b18272c"}
Sep 11 20:28:11.557576 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:11.557198    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7556-d16180" podUID="0ec990cd-5a56-4a82-aa40-08ad37696ae8" containerName="test-curl-7556-d16180" containerID="cri-o://befad8e91c9d2e16b4a9893dff49a3766d275afb6cbca811e7bb9a99deaaca8d" gracePeriod=2
Sep 11 20:28:11.575119 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:11.575078    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7556-545f33" podStartSLOduration=1.575065046 podStartE2EDuration="1.575065046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:28:11.574670499 +0000 UTC m=+1661.566832765" watchObservedRunningTime="2026-09-11 20:28:11.575065046 +0000 UTC m=+1661.567227316"
Sep 11 20:28:12.175917 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:12.175882    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7556-545f33"]
Sep 11 20:28:12.184391 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:12.184353    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7556-545f33"]
Sep 11 20:28:12.562562 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:28:12.562530    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7556-545f33" container="test-curl-7556-545f33"
Sep 11 20:28:13.568033 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:13.567997    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7556-545f33" podUID="927e8641-5ec7-4cf4-abc6-99450a4e02b6" containerName="test-curl-7556-545f33" containerID="cri-o://d55115c7468828213f5d66ab93d68e9ee07a30e3ede8e87e8e1215cc526312dc" gracePeriod=2
Sep 11 20:28:13.703112 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:13.703082    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7556-d16180"
Sep 11 20:28:13.774725 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:13.774696    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68p87\" (UniqueName: \"kubernetes.io/projected/0ec990cd-5a56-4a82-aa40-08ad37696ae8-kube-api-access-68p87\") pod \"0ec990cd-5a56-4a82-aa40-08ad37696ae8\" (UID: \"0ec990cd-5a56-4a82-aa40-08ad37696ae8\") "
Sep 11 20:28:13.776735 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:13.776709    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec990cd-5a56-4a82-aa40-08ad37696ae8-kube-api-access-68p87" (OuterVolumeSpecName: "kube-api-access-68p87") pod "0ec990cd-5a56-4a82-aa40-08ad37696ae8" (UID: "0ec990cd-5a56-4a82-aa40-08ad37696ae8"). InnerVolumeSpecName "kube-api-access-68p87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:28:13.875367 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:13.875288    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68p87\" (UniqueName: \"kubernetes.io/projected/0ec990cd-5a56-4a82-aa40-08ad37696ae8-kube-api-access-68p87\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:28:14.228334 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:14.228295    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"]
Sep 11 20:28:14.440984 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:14.440949    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ec990cd-5a56-4a82-aa40-08ad37696ae8" path="/var/lib/kubelet/pods/0ec990cd-5a56-4a82-aa40-08ad37696ae8/volumes"
Sep 11 20:28:14.574472 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:14.574386    2580 generic.go:356] "Generic (PLEG): container finished" podID="0ec990cd-5a56-4a82-aa40-08ad37696ae8" containerID="befad8e91c9d2e16b4a9893dff49a3766d275afb6cbca811e7bb9a99deaaca8d" exitCode=137
Sep 11 20:28:14.574472 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:14.574428    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7556-d16180"
Sep 11 20:28:14.574947 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:14.574505    2580 scope.go:117] "RemoveContainer" containerID="befad8e91c9d2e16b4a9893dff49a3766d275afb6cbca811e7bb9a99deaaca8d"
Sep 11 20:28:14.583688 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:14.583670    2580 scope.go:117] "RemoveContainer" containerID="befad8e91c9d2e16b4a9893dff49a3766d275afb6cbca811e7bb9a99deaaca8d"
Sep 11 20:28:14.583962 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:28:14.583946    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"befad8e91c9d2e16b4a9893dff49a3766d275afb6cbca811e7bb9a99deaaca8d\": container with ID starting with befad8e91c9d2e16b4a9893dff49a3766d275afb6cbca811e7bb9a99deaaca8d not found: ID does not exist" containerID="befad8e91c9d2e16b4a9893dff49a3766d275afb6cbca811e7bb9a99deaaca8d"
Sep 11 20:28:14.584012 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:14.583970    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"befad8e91c9d2e16b4a9893dff49a3766d275afb6cbca811e7bb9a99deaaca8d"} err="failed to get container status \"befad8e91c9d2e16b4a9893dff49a3766d275afb6cbca811e7bb9a99deaaca8d\": rpc error: code = NotFound desc = could not find container \"befad8e91c9d2e16b4a9893dff49a3766d275afb6cbca811e7bb9a99deaaca8d\": container with ID starting with befad8e91c9d2e16b4a9893dff49a3766d275afb6cbca811e7bb9a99deaaca8d not found: ID does not exist"
Sep 11 20:28:15.206100 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:15.206059    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"]
Sep 11 20:28:15.756979 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:15.756948    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7556-545f33"
Sep 11 20:28:15.890728 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:15.890630    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmhnm\" (UniqueName: \"kubernetes.io/projected/927e8641-5ec7-4cf4-abc6-99450a4e02b6-kube-api-access-dmhnm\") pod \"927e8641-5ec7-4cf4-abc6-99450a4e02b6\" (UID: \"927e8641-5ec7-4cf4-abc6-99450a4e02b6\") "
Sep 11 20:28:15.892722 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:15.892698    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/927e8641-5ec7-4cf4-abc6-99450a4e02b6-kube-api-access-dmhnm" (OuterVolumeSpecName: "kube-api-access-dmhnm") pod "927e8641-5ec7-4cf4-abc6-99450a4e02b6" (UID: "927e8641-5ec7-4cf4-abc6-99450a4e02b6"). InnerVolumeSpecName "kube-api-access-dmhnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:28:15.991712 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:15.991675    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmhnm\" (UniqueName: \"kubernetes.io/projected/927e8641-5ec7-4cf4-abc6-99450a4e02b6-kube-api-access-dmhnm\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:28:16.440924 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:16.440881    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="927e8641-5ec7-4cf4-abc6-99450a4e02b6" path="/var/lib/kubelet/pods/927e8641-5ec7-4cf4-abc6-99450a4e02b6/volumes"
Sep 11 20:28:16.587965 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:16.587933    2580 generic.go:356] "Generic (PLEG): container finished" podID="927e8641-5ec7-4cf4-abc6-99450a4e02b6" containerID="d55115c7468828213f5d66ab93d68e9ee07a30e3ede8e87e8e1215cc526312dc" exitCode=137
Sep 11 20:28:16.588142 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:16.587974    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7556-545f33"
Sep 11 20:28:16.588142 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:16.588005    2580 scope.go:117] "RemoveContainer" containerID="d55115c7468828213f5d66ab93d68e9ee07a30e3ede8e87e8e1215cc526312dc"
Sep 11 20:28:16.596570 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:16.596553    2580 scope.go:117] "RemoveContainer" containerID="d55115c7468828213f5d66ab93d68e9ee07a30e3ede8e87e8e1215cc526312dc"
Sep 11 20:28:16.596863 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:28:16.596843    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d55115c7468828213f5d66ab93d68e9ee07a30e3ede8e87e8e1215cc526312dc\": container with ID starting with d55115c7468828213f5d66ab93d68e9ee07a30e3ede8e87e8e1215cc526312dc not found: ID does not exist" containerID="d55115c7468828213f5d66ab93d68e9ee07a30e3ede8e87e8e1215cc526312dc"
Sep 11 20:28:16.596920 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:16.596869    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d55115c7468828213f5d66ab93d68e9ee07a30e3ede8e87e8e1215cc526312dc"} err="failed to get container status \"d55115c7468828213f5d66ab93d68e9ee07a30e3ede8e87e8e1215cc526312dc\": rpc error: code = NotFound desc = could not find container \"d55115c7468828213f5d66ab93d68e9ee07a30e3ede8e87e8e1215cc526312dc\": container with ID starting with d55115c7468828213f5d66ab93d68e9ee07a30e3ede8e87e8e1215cc526312dc not found: ID does not exist"
Sep 11 20:28:17.829146 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:17.829108    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:28:18.831672 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:18.831640    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"]
Sep 11 20:28:18.832063 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:18.831993    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="927e8641-5ec7-4cf4-abc6-99450a4e02b6" containerName="test-curl-7556-545f33"
Sep 11 20:28:18.832063 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:18.832007    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="927e8641-5ec7-4cf4-abc6-99450a4e02b6" containerName="test-curl-7556-545f33"
Sep 11 20:28:18.832171 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:18.832086    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="927e8641-5ec7-4cf4-abc6-99450a4e02b6" containerName="test-curl-7556-545f33"
Sep 11 20:28:18.836363 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:18.836341    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:28:18.839950 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:18.839930    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-fcb0e4\"/\"rate-test-model-fcb0e4-kserve-self-signed-certs\""
Sep 11 20:28:18.915275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:18.915234    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/805fe33b-b655-44c3-a0dc-e09d3d3e7433-home\") pod \"rate-test-model-fcb0e4-kserve-89ffb57ff-4td94\" (UID: \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:28:18.915461 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:18.915317    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/805fe33b-b655-44c3-a0dc-e09d3d3e7433-model-cache\") pod \"rate-test-model-fcb0e4-kserve-89ffb57ff-4td94\" (UID: \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:28:18.915461 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:18.915373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/805fe33b-b655-44c3-a0dc-e09d3d3e7433-tmp-dir\") pod \"rate-test-model-fcb0e4-kserve-89ffb57ff-4td94\" (UID: \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:28:18.915593 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:18.915525    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/805fe33b-b655-44c3-a0dc-e09d3d3e7433-tls-certs\") pod \"rate-test-model-fcb0e4-kserve-89ffb57ff-4td94\" (UID: \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:28:18.915593 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:18.915565    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxlnt\" (UniqueName: \"kubernetes.io/projected/805fe33b-b655-44c3-a0dc-e09d3d3e7433-kube-api-access-qxlnt\") pod \"rate-test-model-fcb0e4-kserve-89ffb57ff-4td94\" (UID: \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:28:18.915701 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:18.915602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/805fe33b-b655-44c3-a0dc-e09d3d3e7433-dshm\") pod \"rate-test-model-fcb0e4-kserve-89ffb57ff-4td94\" (UID: \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:28:18.918320 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:18.918295    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"]
Sep 11 20:28:19.016391 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:19.016351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/805fe33b-b655-44c3-a0dc-e09d3d3e7433-model-cache\") pod \"rate-test-model-fcb0e4-kserve-89ffb57ff-4td94\" (UID: \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:28:19.016391 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:19.016403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/805fe33b-b655-44c3-a0dc-e09d3d3e7433-tmp-dir\") pod \"rate-test-model-fcb0e4-kserve-89ffb57ff-4td94\" (UID: \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:28:19.016712 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:19.016440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/805fe33b-b655-44c3-a0dc-e09d3d3e7433-tls-certs\") pod \"rate-test-model-fcb0e4-kserve-89ffb57ff-4td94\" (UID: \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:28:19.016712 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:19.016456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxlnt\" (UniqueName: \"kubernetes.io/projected/805fe33b-b655-44c3-a0dc-e09d3d3e7433-kube-api-access-qxlnt\") pod \"rate-test-model-fcb0e4-kserve-89ffb57ff-4td94\" (UID: \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:28:19.016712 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:19.016478    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/805fe33b-b655-44c3-a0dc-e09d3d3e7433-dshm\") pod \"rate-test-model-fcb0e4-kserve-89ffb57ff-4td94\" (UID: \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:28:19.016712 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:19.016564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/805fe33b-b655-44c3-a0dc-e09d3d3e7433-home\") pod \"rate-test-model-fcb0e4-kserve-89ffb57ff-4td94\" (UID: \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:28:19.016914 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:19.016883    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/805fe33b-b655-44c3-a0dc-e09d3d3e7433-model-cache\") pod \"rate-test-model-fcb0e4-kserve-89ffb57ff-4td94\" (UID: \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:28:19.016962 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:19.016919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/805fe33b-b655-44c3-a0dc-e09d3d3e7433-tmp-dir\") pod \"rate-test-model-fcb0e4-kserve-89ffb57ff-4td94\" (UID: \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:28:19.016962 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:19.016934    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/805fe33b-b655-44c3-a0dc-e09d3d3e7433-home\") pod \"rate-test-model-fcb0e4-kserve-89ffb57ff-4td94\" (UID: \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:28:19.018887 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:19.018860    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/805fe33b-b655-44c3-a0dc-e09d3d3e7433-dshm\") pod \"rate-test-model-fcb0e4-kserve-89ffb57ff-4td94\" (UID: \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:28:19.019008 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:19.018976    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/805fe33b-b655-44c3-a0dc-e09d3d3e7433-tls-certs\") pod \"rate-test-model-fcb0e4-kserve-89ffb57ff-4td94\" (UID: \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:28:19.027454 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:19.027433    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxlnt\" (UniqueName: \"kubernetes.io/projected/805fe33b-b655-44c3-a0dc-e09d3d3e7433-kube-api-access-qxlnt\") pod \"rate-test-model-fcb0e4-kserve-89ffb57ff-4td94\" (UID: \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\") " pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:28:19.145718 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:19.145625    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:28:19.423532 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:19.423507    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"]
Sep 11 20:28:19.425220 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:28:19.425194    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod805fe33b_b655_44c3_a0dc_e09d3d3e7433.slice/crio-415b4e9c84fd82b3654c68704d291e578bed9acb740e4eb2ed6669340b98f07e WatchSource:0}: Error finding container 415b4e9c84fd82b3654c68704d291e578bed9acb740e4eb2ed6669340b98f07e: Status 404 returned error can't find the container with id 415b4e9c84fd82b3654c68704d291e578bed9acb740e4eb2ed6669340b98f07e
Sep 11 20:28:19.606415 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:19.606377    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94" event={"ID":"805fe33b-b655-44c3-a0dc-e09d3d3e7433","Type":"ContainerStarted","Data":"9d394f8d328d485972f5d1cf74b88ba7be6bad7c5900e8a3667b0ef799599a45"}
Sep 11 20:28:19.606588 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:19.606426    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94" event={"ID":"805fe33b-b655-44c3-a0dc-e09d3d3e7433","Type":"ContainerStarted","Data":"415b4e9c84fd82b3654c68704d291e578bed9acb740e4eb2ed6669340b98f07e"}
Sep 11 20:28:19.632279 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:19.632234    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94" podStartSLOduration=1.632217755 podStartE2EDuration="1.632217755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:28:19.631662825 +0000 UTC m=+1669.623825094" watchObservedRunningTime="2026-09-11 20:28:19.632217755 +0000 UTC m=+1669.624380003"
Sep 11 20:28:28.213153 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:28.213110    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:28:29.146034 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.145993    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:28:29.146276 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.146255    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:28:29.158585 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.158562    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:28:29.246846 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.246807    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv" podUID="a5fc4658-e8e1-4242-b593-2fdae0e8781a" containerName="main" containerID="cri-o://2b51b7945f40a57740f84e9b679bf973173c3b38aae552252c4f0499ce594177" gracePeriod=45
Sep 11 20:28:29.494126 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.494098    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:28:29.614460 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.614426    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fc4658-e8e1-4242-b593-2fdae0e8781a-model-cache\") pod \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\" (UID: \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\") "
Sep 11 20:28:29.614460 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.614463    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fc4658-e8e1-4242-b593-2fdae0e8781a-dshm\") pod \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\" (UID: \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\") "
Sep 11 20:28:29.614681 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.614533    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fc4658-e8e1-4242-b593-2fdae0e8781a-tls-certs\") pod \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\" (UID: \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\") "
Sep 11 20:28:29.614681 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.614564    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8n4w8\" (UniqueName: \"kubernetes.io/projected/a5fc4658-e8e1-4242-b593-2fdae0e8781a-kube-api-access-8n4w8\") pod \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\" (UID: \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\") "
Sep 11 20:28:29.614681 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.614635    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fc4658-e8e1-4242-b593-2fdae0e8781a-home\") pod \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\" (UID: \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\") "
Sep 11 20:28:29.614681 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.614671    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fc4658-e8e1-4242-b593-2fdae0e8781a-tmp-dir\") pod \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\" (UID: \"a5fc4658-e8e1-4242-b593-2fdae0e8781a\") "
Sep 11 20:28:29.614846 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.614679    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fc4658-e8e1-4242-b593-2fdae0e8781a-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5fc4658-e8e1-4242-b593-2fdae0e8781a" (UID: "a5fc4658-e8e1-4242-b593-2fdae0e8781a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:28:29.614913 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.614880    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fc4658-e8e1-4242-b593-2fdae0e8781a-home" (OuterVolumeSpecName: "home") pod "a5fc4658-e8e1-4242-b593-2fdae0e8781a" (UID: "a5fc4658-e8e1-4242-b593-2fdae0e8781a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:28:29.614972 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.614957    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fc4658-e8e1-4242-b593-2fdae0e8781a-model-cache\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:28:29.615031 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.614975    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fc4658-e8e1-4242-b593-2fdae0e8781a-home\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:28:29.615083 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.615033    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fc4658-e8e1-4242-b593-2fdae0e8781a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5fc4658-e8e1-4242-b593-2fdae0e8781a" (UID: "a5fc4658-e8e1-4242-b593-2fdae0e8781a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:28:29.616739 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.616717    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5fc4658-e8e1-4242-b593-2fdae0e8781a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5fc4658-e8e1-4242-b593-2fdae0e8781a" (UID: "a5fc4658-e8e1-4242-b593-2fdae0e8781a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:28:29.616837 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.616734    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5fc4658-e8e1-4242-b593-2fdae0e8781a-kube-api-access-8n4w8" (OuterVolumeSpecName: "kube-api-access-8n4w8") pod "a5fc4658-e8e1-4242-b593-2fdae0e8781a" (UID: "a5fc4658-e8e1-4242-b593-2fdae0e8781a"). InnerVolumeSpecName "kube-api-access-8n4w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:28:29.616837 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.616765    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fc4658-e8e1-4242-b593-2fdae0e8781a-dshm" (OuterVolumeSpecName: "dshm") pod "a5fc4658-e8e1-4242-b593-2fdae0e8781a" (UID: "a5fc4658-e8e1-4242-b593-2fdae0e8781a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:28:29.661482 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.661451    2580 generic.go:356] "Generic (PLEG): container finished" podID="a5fc4658-e8e1-4242-b593-2fdae0e8781a" containerID="2b51b7945f40a57740f84e9b679bf973173c3b38aae552252c4f0499ce594177" exitCode=0
Sep 11 20:28:29.661692 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.661514    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"
Sep 11 20:28:29.661692 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.661522    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv" event={"ID":"a5fc4658-e8e1-4242-b593-2fdae0e8781a","Type":"ContainerDied","Data":"2b51b7945f40a57740f84e9b679bf973173c3b38aae552252c4f0499ce594177"}
Sep 11 20:28:29.661692 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.661571    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv" event={"ID":"a5fc4658-e8e1-4242-b593-2fdae0e8781a","Type":"ContainerDied","Data":"b9973f985dcb4a97d675ffd636df11992402b18f4b850f3d5fb142fdf5986318"}
Sep 11 20:28:29.661692 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.661590    2580 scope.go:117] "RemoveContainer" containerID="2b51b7945f40a57740f84e9b679bf973173c3b38aae552252c4f0499ce594177"
Sep 11 20:28:29.671324 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.671306    2580 scope.go:117] "RemoveContainer" containerID="2b51b7945f40a57740f84e9b679bf973173c3b38aae552252c4f0499ce594177"
Sep 11 20:28:29.671637 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:28:29.671615    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b51b7945f40a57740f84e9b679bf973173c3b38aae552252c4f0499ce594177\": container with ID starting with 2b51b7945f40a57740f84e9b679bf973173c3b38aae552252c4f0499ce594177 not found: ID does not exist" containerID="2b51b7945f40a57740f84e9b679bf973173c3b38aae552252c4f0499ce594177"
Sep 11 20:28:29.671752 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.671644    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b51b7945f40a57740f84e9b679bf973173c3b38aae552252c4f0499ce594177"} err="failed to get container status \"2b51b7945f40a57740f84e9b679bf973173c3b38aae552252c4f0499ce594177\": rpc error: code = NotFound desc = could not find container \"2b51b7945f40a57740f84e9b679bf973173c3b38aae552252c4f0499ce594177\": container with ID starting with 2b51b7945f40a57740f84e9b679bf973173c3b38aae552252c4f0499ce594177 not found: ID does not exist"
Sep 11 20:28:29.673819 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.673790    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:28:29.694387 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.694361    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"]
Sep 11 20:28:29.696928 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.696908    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcb0e4/sub-test-model-fcb0e4-kserve-595f7ff5f5-bw6dv"]
Sep 11 20:28:29.715593 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.715572    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n4w8\" (UniqueName: \"kubernetes.io/projected/a5fc4658-e8e1-4242-b593-2fdae0e8781a-kube-api-access-8n4w8\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:28:29.715683 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.715594    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fc4658-e8e1-4242-b593-2fdae0e8781a-tmp-dir\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:28:29.715683 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.715608    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fc4658-e8e1-4242-b593-2fdae0e8781a-dshm\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:28:29.715683 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:29.715616    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fc4658-e8e1-4242-b593-2fdae0e8781a-tls-certs\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:28:30.224610 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.224571    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp" podUID="d2df8979-05ee-4a91-a18b-fb448eb789f1" containerName="main" containerID="cri-o://e695d885ff22a8d4e1bcc3cdb7a0e258259ce42401e22081463491a2839ea2f3" gracePeriod=45
Sep 11 20:28:30.448872 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.448836    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5fc4658-e8e1-4242-b593-2fdae0e8781a" path="/var/lib/kubelet/pods/a5fc4658-e8e1-4242-b593-2fdae0e8781a/volumes"
Sep 11 20:28:30.478669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.478611    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:28:30.624824 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.624791    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2df8979-05ee-4a91-a18b-fb448eb789f1-dshm\") pod \"d2df8979-05ee-4a91-a18b-fb448eb789f1\" (UID: \"d2df8979-05ee-4a91-a18b-fb448eb789f1\") "
Sep 11 20:28:30.624824 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.624826    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2df8979-05ee-4a91-a18b-fb448eb789f1-tmp-dir\") pod \"d2df8979-05ee-4a91-a18b-fb448eb789f1\" (UID: \"d2df8979-05ee-4a91-a18b-fb448eb789f1\") "
Sep 11 20:28:30.625018 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.624852    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2df8979-05ee-4a91-a18b-fb448eb789f1-home\") pod \"d2df8979-05ee-4a91-a18b-fb448eb789f1\" (UID: \"d2df8979-05ee-4a91-a18b-fb448eb789f1\") "
Sep 11 20:28:30.625018 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.624908    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2df8979-05ee-4a91-a18b-fb448eb789f1-model-cache\") pod \"d2df8979-05ee-4a91-a18b-fb448eb789f1\" (UID: \"d2df8979-05ee-4a91-a18b-fb448eb789f1\") "
Sep 11 20:28:30.625018 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.624961    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97rl2\" (UniqueName: \"kubernetes.io/projected/d2df8979-05ee-4a91-a18b-fb448eb789f1-kube-api-access-97rl2\") pod \"d2df8979-05ee-4a91-a18b-fb448eb789f1\" (UID: \"d2df8979-05ee-4a91-a18b-fb448eb789f1\") "
Sep 11 20:28:30.625185 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.625019    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2df8979-05ee-4a91-a18b-fb448eb789f1-tls-certs\") pod \"d2df8979-05ee-4a91-a18b-fb448eb789f1\" (UID: \"d2df8979-05ee-4a91-a18b-fb448eb789f1\") "
Sep 11 20:28:30.625185 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.625134    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2df8979-05ee-4a91-a18b-fb448eb789f1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d2df8979-05ee-4a91-a18b-fb448eb789f1" (UID: "d2df8979-05ee-4a91-a18b-fb448eb789f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:28:30.625185 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.625142    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2df8979-05ee-4a91-a18b-fb448eb789f1-home" (OuterVolumeSpecName: "home") pod "d2df8979-05ee-4a91-a18b-fb448eb789f1" (UID: "d2df8979-05ee-4a91-a18b-fb448eb789f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:28:30.625345 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.625179    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2df8979-05ee-4a91-a18b-fb448eb789f1-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2df8979-05ee-4a91-a18b-fb448eb789f1" (UID: "d2df8979-05ee-4a91-a18b-fb448eb789f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:28:30.625472 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.625447    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2df8979-05ee-4a91-a18b-fb448eb789f1-tmp-dir\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:28:30.625564 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.625475    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2df8979-05ee-4a91-a18b-fb448eb789f1-home\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:28:30.625564 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.625511    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2df8979-05ee-4a91-a18b-fb448eb789f1-model-cache\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:28:30.627215 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.627189    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2df8979-05ee-4a91-a18b-fb448eb789f1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2df8979-05ee-4a91-a18b-fb448eb789f1" (UID: "d2df8979-05ee-4a91-a18b-fb448eb789f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:28:30.627487 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.627462    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2df8979-05ee-4a91-a18b-fb448eb789f1-dshm" (OuterVolumeSpecName: "dshm") pod "d2df8979-05ee-4a91-a18b-fb448eb789f1" (UID: "d2df8979-05ee-4a91-a18b-fb448eb789f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:28:30.627600 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.627478    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2df8979-05ee-4a91-a18b-fb448eb789f1-kube-api-access-97rl2" (OuterVolumeSpecName: "kube-api-access-97rl2") pod "d2df8979-05ee-4a91-a18b-fb448eb789f1" (UID: "d2df8979-05ee-4a91-a18b-fb448eb789f1"). InnerVolumeSpecName "kube-api-access-97rl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:28:30.668267 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.668232    2580 generic.go:356] "Generic (PLEG): container finished" podID="d2df8979-05ee-4a91-a18b-fb448eb789f1" containerID="e695d885ff22a8d4e1bcc3cdb7a0e258259ce42401e22081463491a2839ea2f3" exitCode=0
Sep 11 20:28:30.668402 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.668276    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"
Sep 11 20:28:30.668402 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.668329    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp" event={"ID":"d2df8979-05ee-4a91-a18b-fb448eb789f1","Type":"ContainerDied","Data":"e695d885ff22a8d4e1bcc3cdb7a0e258259ce42401e22081463491a2839ea2f3"}
Sep 11 20:28:30.668402 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.668372    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp" event={"ID":"d2df8979-05ee-4a91-a18b-fb448eb789f1","Type":"ContainerDied","Data":"480b52e4ee9dc44fafdd6b9e67ef84ff8f9b65dd1cba95cf53071774994442a0"}
Sep 11 20:28:30.668402 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.668391    2580 scope.go:117] "RemoveContainer" containerID="e695d885ff22a8d4e1bcc3cdb7a0e258259ce42401e22081463491a2839ea2f3"
Sep 11 20:28:30.677734 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.677714    2580 scope.go:117] "RemoveContainer" containerID="e695d885ff22a8d4e1bcc3cdb7a0e258259ce42401e22081463491a2839ea2f3"
Sep 11 20:28:30.677977 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:28:30.677958    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e695d885ff22a8d4e1bcc3cdb7a0e258259ce42401e22081463491a2839ea2f3\": container with ID starting with e695d885ff22a8d4e1bcc3cdb7a0e258259ce42401e22081463491a2839ea2f3 not found: ID does not exist" containerID="e695d885ff22a8d4e1bcc3cdb7a0e258259ce42401e22081463491a2839ea2f3"
Sep 11 20:28:30.678022 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.677987    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e695d885ff22a8d4e1bcc3cdb7a0e258259ce42401e22081463491a2839ea2f3"} err="failed to get container status \"e695d885ff22a8d4e1bcc3cdb7a0e258259ce42401e22081463491a2839ea2f3\": rpc error: code = NotFound desc = could not find container \"e695d885ff22a8d4e1bcc3cdb7a0e258259ce42401e22081463491a2839ea2f3\": container with ID starting with e695d885ff22a8d4e1bcc3cdb7a0e258259ce42401e22081463491a2839ea2f3 not found: ID does not exist"
Sep 11 20:28:30.693439 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.693418    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"]
Sep 11 20:28:30.695702 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.695682    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-092e1a/sub-test-model-092e1a-kserve-769b64d88c-hpsnp"]
Sep 11 20:28:30.726744 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.726723    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2df8979-05ee-4a91-a18b-fb448eb789f1-dshm\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:28:30.726827 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.726745    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97rl2\" (UniqueName: \"kubernetes.io/projected/d2df8979-05ee-4a91-a18b-fb448eb789f1-kube-api-access-97rl2\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:28:30.726827 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.726754    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2df8979-05ee-4a91-a18b-fb448eb789f1-tls-certs\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:28:30.893311 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:30.893236    2580 scope.go:117] "RemoveContainer" containerID="2961ad1536473b474c460ea88a4ad652b3374c941361378b8aea58d92ed12a51"
Sep 11 20:28:32.441114 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:32.441082    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2df8979-05ee-4a91-a18b-fb448eb789f1" path="/var/lib/kubelet/pods/d2df8979-05ee-4a91-a18b-fb448eb789f1/volumes"
Sep 11 20:28:49.313477 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.313445    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"]
Sep 11 20:28:49.313834 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.313811    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2df8979-05ee-4a91-a18b-fb448eb789f1" containerName="main"
Sep 11 20:28:49.313834 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.313823    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2df8979-05ee-4a91-a18b-fb448eb789f1" containerName="main"
Sep 11 20:28:49.313834 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.313832    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5fc4658-e8e1-4242-b593-2fdae0e8781a" containerName="main"
Sep 11 20:28:49.313934 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.313838    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5fc4658-e8e1-4242-b593-2fdae0e8781a" containerName="main"
Sep 11 20:28:49.313934 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.313895    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5fc4658-e8e1-4242-b593-2fdae0e8781a" containerName="main"
Sep 11 20:28:49.313934 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.313905    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2df8979-05ee-4a91-a18b-fb448eb789f1" containerName="main"
Sep 11 20:28:49.316991 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.316973    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:28:49.321708 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.321683    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-092e1a\"/\"rate-test-model-092e1a-kserve-self-signed-certs\""
Sep 11 20:28:49.321824 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.321683    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-092e1a\"/\"default-dockercfg-2slpb\""
Sep 11 20:28:49.321824 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.321688    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-092e1a\"/\"kube-root-ca.crt\""
Sep 11 20:28:49.321824 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.321742    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-092e1a\"/\"openshift-service-ca.crt\""
Sep 11 20:28:49.327415 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.327393    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"]
Sep 11 20:28:49.388789 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.388763    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f915185-60fb-45b8-80be-85879abc7aeb-home\") pod \"rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c\" (UID: \"7f915185-60fb-45b8-80be-85879abc7aeb\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:28:49.388919 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.388793    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f915185-60fb-45b8-80be-85879abc7aeb-tls-certs\") pod \"rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c\" (UID: \"7f915185-60fb-45b8-80be-85879abc7aeb\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:28:49.388919 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.388812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f915185-60fb-45b8-80be-85879abc7aeb-model-cache\") pod \"rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c\" (UID: \"7f915185-60fb-45b8-80be-85879abc7aeb\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:28:49.388919 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.388828    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f915185-60fb-45b8-80be-85879abc7aeb-dshm\") pod \"rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c\" (UID: \"7f915185-60fb-45b8-80be-85879abc7aeb\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:28:49.388919 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.388877    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f915185-60fb-45b8-80be-85879abc7aeb-tmp-dir\") pod \"rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c\" (UID: \"7f915185-60fb-45b8-80be-85879abc7aeb\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:28:49.389055 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.388927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr226\" (UniqueName: \"kubernetes.io/projected/7f915185-60fb-45b8-80be-85879abc7aeb-kube-api-access-nr226\") pod \"rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c\" (UID: \"7f915185-60fb-45b8-80be-85879abc7aeb\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:28:49.489369 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.489337    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr226\" (UniqueName: \"kubernetes.io/projected/7f915185-60fb-45b8-80be-85879abc7aeb-kube-api-access-nr226\") pod \"rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c\" (UID: \"7f915185-60fb-45b8-80be-85879abc7aeb\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:28:49.489573 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.489400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f915185-60fb-45b8-80be-85879abc7aeb-home\") pod \"rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c\" (UID: \"7f915185-60fb-45b8-80be-85879abc7aeb\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:28:49.489573 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.489432    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f915185-60fb-45b8-80be-85879abc7aeb-tls-certs\") pod \"rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c\" (UID: \"7f915185-60fb-45b8-80be-85879abc7aeb\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:28:49.489573 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.489456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f915185-60fb-45b8-80be-85879abc7aeb-model-cache\") pod \"rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c\" (UID: \"7f915185-60fb-45b8-80be-85879abc7aeb\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:28:49.489573 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.489479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f915185-60fb-45b8-80be-85879abc7aeb-dshm\") pod \"rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c\" (UID: \"7f915185-60fb-45b8-80be-85879abc7aeb\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:28:49.489755 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.489614    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f915185-60fb-45b8-80be-85879abc7aeb-tmp-dir\") pod \"rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c\" (UID: \"7f915185-60fb-45b8-80be-85879abc7aeb\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:28:49.489852 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.489833    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f915185-60fb-45b8-80be-85879abc7aeb-model-cache\") pod \"rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c\" (UID: \"7f915185-60fb-45b8-80be-85879abc7aeb\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:28:49.489917 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.489865    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f915185-60fb-45b8-80be-85879abc7aeb-home\") pod \"rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c\" (UID: \"7f915185-60fb-45b8-80be-85879abc7aeb\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:28:49.490021 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.490001    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f915185-60fb-45b8-80be-85879abc7aeb-tmp-dir\") pod \"rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c\" (UID: \"7f915185-60fb-45b8-80be-85879abc7aeb\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:28:49.491846 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.491825    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f915185-60fb-45b8-80be-85879abc7aeb-dshm\") pod \"rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c\" (UID: \"7f915185-60fb-45b8-80be-85879abc7aeb\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:28:49.492098 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.492082    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f915185-60fb-45b8-80be-85879abc7aeb-tls-certs\") pod \"rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c\" (UID: \"7f915185-60fb-45b8-80be-85879abc7aeb\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:28:49.498697 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.498673    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr226\" (UniqueName: \"kubernetes.io/projected/7f915185-60fb-45b8-80be-85879abc7aeb-kube-api-access-nr226\") pod \"rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c\" (UID: \"7f915185-60fb-45b8-80be-85879abc7aeb\") " pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:28:49.626694 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.626615    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:28:49.905905 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:49.905878    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"]
Sep 11 20:28:49.907339 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:28:49.907311    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f915185_60fb_45b8_80be_85879abc7aeb.slice/crio-2343414d52d352c538271f02246413102432e2b9d5468195c090cebacebf7740 WatchSource:0}: Error finding container 2343414d52d352c538271f02246413102432e2b9d5468195c090cebacebf7740: Status 404 returned error can't find the container with id 2343414d52d352c538271f02246413102432e2b9d5468195c090cebacebf7740
Sep 11 20:28:50.777217 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:50.777178    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c" event={"ID":"7f915185-60fb-45b8-80be-85879abc7aeb","Type":"ContainerStarted","Data":"309dc9645fdddecc49729150dbcd200c82c956461c742c1190c532e49acdaacf"}
Sep 11 20:28:50.777217 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:50.777222    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c" event={"ID":"7f915185-60fb-45b8-80be-85879abc7aeb","Type":"ContainerStarted","Data":"2343414d52d352c538271f02246413102432e2b9d5468195c090cebacebf7740"}
Sep 11 20:28:50.799802 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:50.799758    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c" podStartSLOduration=1.799744446 podStartE2EDuration="1.799744446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:28:50.797936078 +0000 UTC m=+1700.790098360" watchObservedRunningTime="2026-09-11 20:28:50.799744446 +0000 UTC m=+1700.791906716"
Sep 11 20:28:52.731917 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:52.731884    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:28:59.627158 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:59.627120    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:28:59.627696 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:59.627419    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:28:59.640010 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:59.639984    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:28:59.835850 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:28:59.835818    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:29:18.021638 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:18.021602    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:29:27.124318 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:27.124286    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:29:33.976875 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:33.976843    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"]
Sep 11 20:29:34.917393 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:34.917359    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"]
Sep 11 20:29:37.836615 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:37.836582    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:29:41.528541 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:41.528506    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p"]
Sep 11 20:29:41.531949 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:41.531929    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p"
Sep 11 20:29:41.535117 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:41.535097    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-248ab6-dockercfg-nvjrg\""
Sep 11 20:29:41.535244 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:41.535141    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-248ab6\""
Sep 11 20:29:41.546420 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:41.546386    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p"]
Sep 11 20:29:41.653480 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:41.653439    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmdnl\" (UniqueName: \"kubernetes.io/projected/20c06bbc-a74c-49b7-aa9e-7d02bb5576c5-kube-api-access-kmdnl\") pod \"payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p\" (UID: \"20c06bbc-a74c-49b7-aa9e-7d02bb5576c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p"
Sep 11 20:29:41.653683 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:41.653535    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c06bbc-a74c-49b7-aa9e-7d02bb5576c5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p\" (UID: \"20c06bbc-a74c-49b7-aa9e-7d02bb5576c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p"
Sep 11 20:29:41.754271 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:41.754234    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmdnl\" (UniqueName: \"kubernetes.io/projected/20c06bbc-a74c-49b7-aa9e-7d02bb5576c5-kube-api-access-kmdnl\") pod \"payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p\" (UID: \"20c06bbc-a74c-49b7-aa9e-7d02bb5576c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p"
Sep 11 20:29:41.754431 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:41.754290    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c06bbc-a74c-49b7-aa9e-7d02bb5576c5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p\" (UID: \"20c06bbc-a74c-49b7-aa9e-7d02bb5576c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p"
Sep 11 20:29:41.754971 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:41.754952    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c06bbc-a74c-49b7-aa9e-7d02bb5576c5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p\" (UID: \"20c06bbc-a74c-49b7-aa9e-7d02bb5576c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p"
Sep 11 20:29:41.764253 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:41.764231    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmdnl\" (UniqueName: \"kubernetes.io/projected/20c06bbc-a74c-49b7-aa9e-7d02bb5576c5-kube-api-access-kmdnl\") pod \"payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p\" (UID: \"20c06bbc-a74c-49b7-aa9e-7d02bb5576c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p"
Sep 11 20:29:41.842264 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:41.842191    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p"
Sep 11 20:29:42.115685 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:42.115657    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p"]
Sep 11 20:29:42.117624 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:29:42.117598    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c06bbc_a74c_49b7_aa9e_7d02bb5576c5.slice/crio-3f4635c140ba199640772292445224e6144a1ac2f3caad80df32d5125540a2f1 WatchSource:0}: Error finding container 3f4635c140ba199640772292445224e6144a1ac2f3caad80df32d5125540a2f1: Status 404 returned error can't find the container with id 3f4635c140ba199640772292445224e6144a1ac2f3caad80df32d5125540a2f1
Sep 11 20:29:43.060973 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:43.060936    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p" event={"ID":"20c06bbc-a74c-49b7-aa9e-7d02bb5576c5","Type":"ContainerStarted","Data":"66d070a22eb7f6837fd66555fb4a87fce3377e12f6980542083b45034b836bbf"}
Sep 11 20:29:43.061359 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:43.061005    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p"
Sep 11 20:29:43.061359 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:43.061033    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p" event={"ID":"20c06bbc-a74c-49b7-aa9e-7d02bb5576c5","Type":"ContainerStarted","Data":"3f4635c140ba199640772292445224e6144a1ac2f3caad80df32d5125540a2f1"}
Sep 11 20:29:43.082805 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:43.082754    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p" podStartSLOduration=2.082740202 podStartE2EDuration="2.082740202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:29:43.080245706 +0000 UTC m=+1753.072407976" watchObservedRunningTime="2026-09-11 20:29:43.082740202 +0000 UTC m=+1753.074902471"
Sep 11 20:29:49.030566 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:49.030488    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94" podUID="805fe33b-b655-44c3-a0dc-e09d3d3e7433" containerName="main" containerID="cri-o://9d394f8d328d485972f5d1cf74b88ba7be6bad7c5900e8a3667b0ef799599a45" gracePeriod=45
Sep 11 20:29:49.282132 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:49.282071    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:29:49.422122 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:49.422086    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/805fe33b-b655-44c3-a0dc-e09d3d3e7433-tls-certs\") pod \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\" (UID: \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\") "
Sep 11 20:29:49.422304 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:49.422140    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxlnt\" (UniqueName: \"kubernetes.io/projected/805fe33b-b655-44c3-a0dc-e09d3d3e7433-kube-api-access-qxlnt\") pod \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\" (UID: \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\") "
Sep 11 20:29:49.422304 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:49.422171    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/805fe33b-b655-44c3-a0dc-e09d3d3e7433-tmp-dir\") pod \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\" (UID: \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\") "
Sep 11 20:29:49.422304 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:49.422222    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/805fe33b-b655-44c3-a0dc-e09d3d3e7433-dshm\") pod \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\" (UID: \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\") "
Sep 11 20:29:49.422304 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:49.422237    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/805fe33b-b655-44c3-a0dc-e09d3d3e7433-model-cache\") pod \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\" (UID: \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\") "
Sep 11 20:29:49.422304 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:49.422253    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/805fe33b-b655-44c3-a0dc-e09d3d3e7433-home\") pod \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\" (UID: \"805fe33b-b655-44c3-a0dc-e09d3d3e7433\") "
Sep 11 20:29:49.422613 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:49.422518    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805fe33b-b655-44c3-a0dc-e09d3d3e7433-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "805fe33b-b655-44c3-a0dc-e09d3d3e7433" (UID: "805fe33b-b655-44c3-a0dc-e09d3d3e7433"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:29:49.422613 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:49.422518    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805fe33b-b655-44c3-a0dc-e09d3d3e7433-model-cache" (OuterVolumeSpecName: "model-cache") pod "805fe33b-b655-44c3-a0dc-e09d3d3e7433" (UID: "805fe33b-b655-44c3-a0dc-e09d3d3e7433"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:29:49.422613 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:49.422586    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805fe33b-b655-44c3-a0dc-e09d3d3e7433-home" (OuterVolumeSpecName: "home") pod "805fe33b-b655-44c3-a0dc-e09d3d3e7433" (UID: "805fe33b-b655-44c3-a0dc-e09d3d3e7433"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:29:49.424331 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:49.424305    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805fe33b-b655-44c3-a0dc-e09d3d3e7433-dshm" (OuterVolumeSpecName: "dshm") pod "805fe33b-b655-44c3-a0dc-e09d3d3e7433" (UID: "805fe33b-b655-44c3-a0dc-e09d3d3e7433"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:29:49.424426 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:49.424335    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805fe33b-b655-44c3-a0dc-e09d3d3e7433-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "805fe33b-b655-44c3-a0dc-e09d3d3e7433" (UID: "805fe33b-b655-44c3-a0dc-e09d3d3e7433"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:29:49.424479 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:49.424438    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805fe33b-b655-44c3-a0dc-e09d3d3e7433-kube-api-access-qxlnt" (OuterVolumeSpecName: "kube-api-access-qxlnt") pod "805fe33b-b655-44c3-a0dc-e09d3d3e7433" (UID: "805fe33b-b655-44c3-a0dc-e09d3d3e7433"). InnerVolumeSpecName "kube-api-access-qxlnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:29:49.523835 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:49.523805    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/805fe33b-b655-44c3-a0dc-e09d3d3e7433-tls-certs\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:29:49.523835 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:49.523831    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxlnt\" (UniqueName: \"kubernetes.io/projected/805fe33b-b655-44c3-a0dc-e09d3d3e7433-kube-api-access-qxlnt\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:29:49.523835 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:49.523841    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/805fe33b-b655-44c3-a0dc-e09d3d3e7433-tmp-dir\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:29:49.524039 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:49.523851    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/805fe33b-b655-44c3-a0dc-e09d3d3e7433-dshm\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:29:49.524039 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:49.523859    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/805fe33b-b655-44c3-a0dc-e09d3d3e7433-model-cache\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:29:49.524039 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:49.523867    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/805fe33b-b655-44c3-a0dc-e09d3d3e7433-home\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:29:49.936653 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:49.936615    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c" podUID="7f915185-60fb-45b8-80be-85879abc7aeb" containerName="main" containerID="cri-o://309dc9645fdddecc49729150dbcd200c82c956461c742c1190c532e49acdaacf" gracePeriod=45
Sep 11 20:29:50.098439 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.098399    2580 generic.go:356] "Generic (PLEG): container finished" podID="805fe33b-b655-44c3-a0dc-e09d3d3e7433" containerID="9d394f8d328d485972f5d1cf74b88ba7be6bad7c5900e8a3667b0ef799599a45" exitCode=0
Sep 11 20:29:50.098909 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.098441    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"
Sep 11 20:29:50.098909 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.098475    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94" event={"ID":"805fe33b-b655-44c3-a0dc-e09d3d3e7433","Type":"ContainerDied","Data":"9d394f8d328d485972f5d1cf74b88ba7be6bad7c5900e8a3667b0ef799599a45"}
Sep 11 20:29:50.098909 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.098540    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94" event={"ID":"805fe33b-b655-44c3-a0dc-e09d3d3e7433","Type":"ContainerDied","Data":"415b4e9c84fd82b3654c68704d291e578bed9acb740e4eb2ed6669340b98f07e"}
Sep 11 20:29:50.098909 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.098558    2580 scope.go:117] "RemoveContainer" containerID="9d394f8d328d485972f5d1cf74b88ba7be6bad7c5900e8a3667b0ef799599a45"
Sep 11 20:29:50.100585 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.100483    2580 generic.go:356] "Generic (PLEG): container finished" podID="7f915185-60fb-45b8-80be-85879abc7aeb" containerID="309dc9645fdddecc49729150dbcd200c82c956461c742c1190c532e49acdaacf" exitCode=0
Sep 11 20:29:50.100585 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.100519    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c" event={"ID":"7f915185-60fb-45b8-80be-85879abc7aeb","Type":"ContainerDied","Data":"309dc9645fdddecc49729150dbcd200c82c956461c742c1190c532e49acdaacf"}
Sep 11 20:29:50.109524 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.109484    2580 scope.go:117] "RemoveContainer" containerID="9d394f8d328d485972f5d1cf74b88ba7be6bad7c5900e8a3667b0ef799599a45"
Sep 11 20:29:50.109827 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:29:50.109801    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d394f8d328d485972f5d1cf74b88ba7be6bad7c5900e8a3667b0ef799599a45\": container with ID starting with 9d394f8d328d485972f5d1cf74b88ba7be6bad7c5900e8a3667b0ef799599a45 not found: ID does not exist" containerID="9d394f8d328d485972f5d1cf74b88ba7be6bad7c5900e8a3667b0ef799599a45"
Sep 11 20:29:50.109892 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.109840    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d394f8d328d485972f5d1cf74b88ba7be6bad7c5900e8a3667b0ef799599a45"} err="failed to get container status \"9d394f8d328d485972f5d1cf74b88ba7be6bad7c5900e8a3667b0ef799599a45\": rpc error: code = NotFound desc = could not find container \"9d394f8d328d485972f5d1cf74b88ba7be6bad7c5900e8a3667b0ef799599a45\": container with ID starting with 9d394f8d328d485972f5d1cf74b88ba7be6bad7c5900e8a3667b0ef799599a45 not found: ID does not exist"
Sep 11 20:29:50.125923 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.125897    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"]
Sep 11 20:29:50.128071 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.128046    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fcb0e4/rate-test-model-fcb0e4-kserve-89ffb57ff-4td94"]
Sep 11 20:29:50.184098 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.184074    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:29:50.331142 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.331054    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f915185-60fb-45b8-80be-85879abc7aeb-home\") pod \"7f915185-60fb-45b8-80be-85879abc7aeb\" (UID: \"7f915185-60fb-45b8-80be-85879abc7aeb\") "
Sep 11 20:29:50.331142 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.331117    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f915185-60fb-45b8-80be-85879abc7aeb-dshm\") pod \"7f915185-60fb-45b8-80be-85879abc7aeb\" (UID: \"7f915185-60fb-45b8-80be-85879abc7aeb\") "
Sep 11 20:29:50.331142 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.331137    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f915185-60fb-45b8-80be-85879abc7aeb-model-cache\") pod \"7f915185-60fb-45b8-80be-85879abc7aeb\" (UID: \"7f915185-60fb-45b8-80be-85879abc7aeb\") "
Sep 11 20:29:50.331385 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.331189    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f915185-60fb-45b8-80be-85879abc7aeb-tls-certs\") pod \"7f915185-60fb-45b8-80be-85879abc7aeb\" (UID: \"7f915185-60fb-45b8-80be-85879abc7aeb\") "
Sep 11 20:29:50.331385 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.331239    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f915185-60fb-45b8-80be-85879abc7aeb-tmp-dir\") pod \"7f915185-60fb-45b8-80be-85879abc7aeb\" (UID: \"7f915185-60fb-45b8-80be-85879abc7aeb\") "
Sep 11 20:29:50.331385 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.331257    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nr226\" (UniqueName: \"kubernetes.io/projected/7f915185-60fb-45b8-80be-85879abc7aeb-kube-api-access-nr226\") pod \"7f915185-60fb-45b8-80be-85879abc7aeb\" (UID: \"7f915185-60fb-45b8-80be-85879abc7aeb\") "
Sep 11 20:29:50.331385 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.331366    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f915185-60fb-45b8-80be-85879abc7aeb-home" (OuterVolumeSpecName: "home") pod "7f915185-60fb-45b8-80be-85879abc7aeb" (UID: "7f915185-60fb-45b8-80be-85879abc7aeb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:29:50.331637 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.331383    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f915185-60fb-45b8-80be-85879abc7aeb-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f915185-60fb-45b8-80be-85879abc7aeb" (UID: "7f915185-60fb-45b8-80be-85879abc7aeb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:29:50.331637 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.331551    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f915185-60fb-45b8-80be-85879abc7aeb-home\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:29:50.331637 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.331566    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f915185-60fb-45b8-80be-85879abc7aeb-model-cache\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:29:50.331637 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.331574    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f915185-60fb-45b8-80be-85879abc7aeb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f915185-60fb-45b8-80be-85879abc7aeb" (UID: "7f915185-60fb-45b8-80be-85879abc7aeb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:29:50.333227 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.333197    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f915185-60fb-45b8-80be-85879abc7aeb-dshm" (OuterVolumeSpecName: "dshm") pod "7f915185-60fb-45b8-80be-85879abc7aeb" (UID: "7f915185-60fb-45b8-80be-85879abc7aeb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:29:50.333374 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.333353    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f915185-60fb-45b8-80be-85879abc7aeb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f915185-60fb-45b8-80be-85879abc7aeb" (UID: "7f915185-60fb-45b8-80be-85879abc7aeb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:29:50.333435 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.333377    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f915185-60fb-45b8-80be-85879abc7aeb-kube-api-access-nr226" (OuterVolumeSpecName: "kube-api-access-nr226") pod "7f915185-60fb-45b8-80be-85879abc7aeb" (UID: "7f915185-60fb-45b8-80be-85879abc7aeb"). InnerVolumeSpecName "kube-api-access-nr226". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:29:50.432885 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.432849    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f915185-60fb-45b8-80be-85879abc7aeb-tls-certs\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:29:50.432885 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.432879    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f915185-60fb-45b8-80be-85879abc7aeb-tmp-dir\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:29:50.432885 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.432887    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr226\" (UniqueName: \"kubernetes.io/projected/7f915185-60fb-45b8-80be-85879abc7aeb-kube-api-access-nr226\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:29:50.433111 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.432896    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f915185-60fb-45b8-80be-85879abc7aeb-dshm\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:29:50.442602 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:50.442570    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="805fe33b-b655-44c3-a0dc-e09d3d3e7433" path="/var/lib/kubelet/pods/805fe33b-b655-44c3-a0dc-e09d3d3e7433/volumes"
Sep 11 20:29:51.108327 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:51.108287    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c" event={"ID":"7f915185-60fb-45b8-80be-85879abc7aeb","Type":"ContainerDied","Data":"2343414d52d352c538271f02246413102432e2b9d5468195c090cebacebf7740"}
Sep 11 20:29:51.108747 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:51.108338    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"
Sep 11 20:29:51.108747 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:51.108349    2580 scope.go:117] "RemoveContainer" containerID="309dc9645fdddecc49729150dbcd200c82c956461c742c1190c532e49acdaacf"
Sep 11 20:29:51.147457 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:51.147430    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"]
Sep 11 20:29:51.155704 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:51.155676    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-092e1a/rate-test-model-092e1a-kserve-5cbffdb6c5-t2r9c"]
Sep 11 20:29:52.440484 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:52.440442    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f915185-60fb-45b8-80be-85879abc7aeb" path="/var/lib/kubelet/pods/7f915185-60fb-45b8-80be-85879abc7aeb/volumes"
Sep 11 20:29:53.428864 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:53.428833    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:29:54.067917 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:29:54.067889    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p"
Sep 11 20:30:00.134593 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:00.134551    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819310-rtwfk"]
Sep 11 20:30:00.135068 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:00.135052    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f915185-60fb-45b8-80be-85879abc7aeb" containerName="main"
Sep 11 20:30:00.135146 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:00.135072    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f915185-60fb-45b8-80be-85879abc7aeb" containerName="main"
Sep 11 20:30:00.135146 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:00.135099    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="805fe33b-b655-44c3-a0dc-e09d3d3e7433" containerName="main"
Sep 11 20:30:00.135146 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:00.135108    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="805fe33b-b655-44c3-a0dc-e09d3d3e7433" containerName="main"
Sep 11 20:30:00.135301 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:00.135207    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="805fe33b-b655-44c3-a0dc-e09d3d3e7433" containerName="main"
Sep 11 20:30:00.135301 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:00.135222    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f915185-60fb-45b8-80be-85879abc7aeb" containerName="main"
Sep 11 20:30:00.139719 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:00.139697    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819310-rtwfk"
Sep 11 20:30:00.143223 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:00.143199    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fcb0e4-29819310-r95pm"]
Sep 11 20:30:00.144258 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:00.144199    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7dbs5\""
Sep 11 20:30:00.146976 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:00.146956    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-092e1a-29819310-6cdwt"]
Sep 11 20:30:00.148781 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:00.148761    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fcb0e4-29819310-r95pm"
Sep 11 20:30:00.150664 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:00.150644    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819310-rtwfk"]
Sep 11 20:30:00.150772 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:00.150716    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-092e1a-29819310-6cdwt"
Sep 11 20:30:00.153755 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:00.153734    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819310-rtwfk"
Sep 11 20:30:00.158624 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:00.157313    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fcb0e4-29819310-r95pm"]
Sep 11 20:30:00.166230 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:00.166213    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fcb0e4-29819310-r95pm"
Sep 11 20:30:00.175596 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:00.175570    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-092e1a-29819310-6cdwt"]
Sep 11 20:30:00.202363 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:00.202328    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-092e1a-29819310-6cdwt"
Sep 11 20:30:00.447659 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:00.447634    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819310-rtwfk"]
Sep 11 20:30:00.450084 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:30:00.450058    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod411174cb_1485_4333_ab49_ad3dc2ec2196.slice/crio-68c3cba58ffa9cb9cb9c5fb89b7701b6264d9f49986440804fcda06d46e58c9b WatchSource:0}: Error finding container 68c3cba58ffa9cb9cb9c5fb89b7701b6264d9f49986440804fcda06d46e58c9b: Status 404 returned error can't find the container with id 68c3cba58ffa9cb9cb9c5fb89b7701b6264d9f49986440804fcda06d46e58c9b
Sep 11 20:30:00.450409 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:00.450386    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fcb0e4-29819310-r95pm"]
Sep 11 20:30:00.451181 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:30:00.451158    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcebaf46_aab1_4b87_b80d_d1517576b19a.slice/crio-858a5e15350c2352b5f25df0042011d09f3e378deed3d2bb8ddb37639d35c5b8 WatchSource:0}: Error finding container 858a5e15350c2352b5f25df0042011d09f3e378deed3d2bb8ddb37639d35c5b8: Status 404 returned error can't find the container with id 858a5e15350c2352b5f25df0042011d09f3e378deed3d2bb8ddb37639d35c5b8
Sep 11 20:30:00.500949 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:00.500922    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-092e1a-29819310-6cdwt"]
Sep 11 20:30:00.502077 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:30:00.502049    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b15257a_7c74_4b1d_a88e_4cbe198dd569.slice/crio-3d3c21db6743aa8ea6363e141876eb31850ca08463a099be3339159528f80d2f WatchSource:0}: Error finding container 3d3c21db6743aa8ea6363e141876eb31850ca08463a099be3339159528f80d2f: Status 404 returned error can't find the container with id 3d3c21db6743aa8ea6363e141876eb31850ca08463a099be3339159528f80d2f
Sep 11 20:30:01.169929 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:01.169829    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-092e1a-29819310-6cdwt" event={"ID":"2b15257a-7c74-4b1d-a88e-4cbe198dd569","Type":"ContainerStarted","Data":"c169be4b91ec8dbb0452f06ec1693f626b9049b82d85e2281e17f654e08fa8d0"}
Sep 11 20:30:01.170329 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:01.169941    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-092e1a-29819310-6cdwt" event={"ID":"2b15257a-7c74-4b1d-a88e-4cbe198dd569","Type":"ContainerStarted","Data":"3d3c21db6743aa8ea6363e141876eb31850ca08463a099be3339159528f80d2f"}
Sep 11 20:30:01.171381 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:01.171355    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fcb0e4-29819310-r95pm" event={"ID":"dcebaf46-aab1-4b87-b80d-d1517576b19a","Type":"ContainerStarted","Data":"d089cf3f0ccf4103e5d31f3d95610cec2fe3204d570906b770544eb9757cbe4d"}
Sep 11 20:30:01.171502 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:01.171405    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fcb0e4-29819310-r95pm" event={"ID":"dcebaf46-aab1-4b87-b80d-d1517576b19a","Type":"ContainerStarted","Data":"858a5e15350c2352b5f25df0042011d09f3e378deed3d2bb8ddb37639d35c5b8"}
Sep 11 20:30:01.172885 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:01.172855    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819310-rtwfk" event={"ID":"411174cb-1485-4333-ab49-ad3dc2ec2196","Type":"ContainerStarted","Data":"733caf5cc8a7f3fed983350f18dba23cd45d65ca8845e80f8ac389423eddefd8"}
Sep 11 20:30:01.172983 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:01.172910    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819310-rtwfk" event={"ID":"411174cb-1485-4333-ab49-ad3dc2ec2196","Type":"ContainerStarted","Data":"68c3cba58ffa9cb9cb9c5fb89b7701b6264d9f49986440804fcda06d46e58c9b"}
Sep 11 20:30:01.186250 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:01.186206    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-092e1a-29819310-6cdwt" podStartSLOduration=1.186192438 podStartE2EDuration="1.186192438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:30:01.184995804 +0000 UTC m=+1771.177158075" watchObservedRunningTime="2026-09-11 20:30:01.186192438 +0000 UTC m=+1771.178354704"
Sep 11 20:30:01.201966 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:01.201922    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fcb0e4-29819310-r95pm" podStartSLOduration=1.201908727 podStartE2EDuration="1.201908727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:30:01.200578466 +0000 UTC m=+1771.192740736" watchObservedRunningTime="2026-09-11 20:30:01.201908727 +0000 UTC m=+1771.194071041"
Sep 11 20:30:01.216610 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:01.216572    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819310-rtwfk" podStartSLOduration=1.216558831 podStartE2EDuration="1.216558831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:30:01.215477077 +0000 UTC m=+1771.207639346" watchObservedRunningTime="2026-09-11 20:30:01.216558831 +0000 UTC m=+1771.208721101"
Sep 11 20:30:02.179949 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:02.179918    2580 generic.go:356] "Generic (PLEG): container finished" podID="2b15257a-7c74-4b1d-a88e-4cbe198dd569" containerID="c169be4b91ec8dbb0452f06ec1693f626b9049b82d85e2281e17f654e08fa8d0" exitCode=0
Sep 11 20:30:02.180358 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:02.180009    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-092e1a-29819310-6cdwt" event={"ID":"2b15257a-7c74-4b1d-a88e-4cbe198dd569","Type":"ContainerDied","Data":"c169be4b91ec8dbb0452f06ec1693f626b9049b82d85e2281e17f654e08fa8d0"}
Sep 11 20:30:02.181632 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:02.181610    2580 generic.go:356] "Generic (PLEG): container finished" podID="dcebaf46-aab1-4b87-b80d-d1517576b19a" containerID="d089cf3f0ccf4103e5d31f3d95610cec2fe3204d570906b770544eb9757cbe4d" exitCode=0
Sep 11 20:30:02.181749 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:02.181685    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fcb0e4-29819310-r95pm" event={"ID":"dcebaf46-aab1-4b87-b80d-d1517576b19a","Type":"ContainerDied","Data":"d089cf3f0ccf4103e5d31f3d95610cec2fe3204d570906b770544eb9757cbe4d"}
Sep 11 20:30:02.183028 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:02.183010    2580 generic.go:356] "Generic (PLEG): container finished" podID="411174cb-1485-4333-ab49-ad3dc2ec2196" containerID="733caf5cc8a7f3fed983350f18dba23cd45d65ca8845e80f8ac389423eddefd8" exitCode=0
Sep 11 20:30:02.183118 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:02.183093    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819310-rtwfk" event={"ID":"411174cb-1485-4333-ab49-ad3dc2ec2196","Type":"ContainerDied","Data":"733caf5cc8a7f3fed983350f18dba23cd45d65ca8845e80f8ac389423eddefd8"}
Sep 11 20:30:03.334848 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:03.334821    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-092e1a-29819310-6cdwt"
Sep 11 20:30:03.371551 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:03.371523    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819310-rtwfk"
Sep 11 20:30:03.374315 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:03.374292    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fcb0e4-29819310-r95pm"
Sep 11 20:30:04.196233 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:04.196196    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-092e1a-29819310-6cdwt"
Sep 11 20:30:04.196444 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:04.196239    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-092e1a-29819310-6cdwt" event={"ID":"2b15257a-7c74-4b1d-a88e-4cbe198dd569","Type":"ContainerDied","Data":"3d3c21db6743aa8ea6363e141876eb31850ca08463a099be3339159528f80d2f"}
Sep 11 20:30:04.196444 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:04.196293    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d3c21db6743aa8ea6363e141876eb31850ca08463a099be3339159528f80d2f"
Sep 11 20:30:04.197850 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:04.197818    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fcb0e4-29819310-r95pm"
Sep 11 20:30:04.198107 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:04.197832    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fcb0e4-29819310-r95pm" event={"ID":"dcebaf46-aab1-4b87-b80d-d1517576b19a","Type":"ContainerDied","Data":"858a5e15350c2352b5f25df0042011d09f3e378deed3d2bb8ddb37639d35c5b8"}
Sep 11 20:30:04.198226 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:04.198150    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="858a5e15350c2352b5f25df0042011d09f3e378deed3d2bb8ddb37639d35c5b8"
Sep 11 20:30:04.199182 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:04.199166    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819310-rtwfk"
Sep 11 20:30:04.199275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:04.199174    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819310-rtwfk" event={"ID":"411174cb-1485-4333-ab49-ad3dc2ec2196","Type":"ContainerDied","Data":"68c3cba58ffa9cb9cb9c5fb89b7701b6264d9f49986440804fcda06d46e58c9b"}
Sep 11 20:30:04.199275 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:04.199226    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68c3cba58ffa9cb9cb9c5fb89b7701b6264d9f49986440804fcda06d46e58c9b"
Sep 11 20:30:16.031403 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:16.031368    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g"]
Sep 11 20:30:16.031858 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:16.031719    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcebaf46-aab1-4b87-b80d-d1517576b19a" containerName="cleanup"
Sep 11 20:30:16.031858 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:16.031731    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcebaf46-aab1-4b87-b80d-d1517576b19a" containerName="cleanup"
Sep 11 20:30:16.031858 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:16.031738    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="411174cb-1485-4333-ab49-ad3dc2ec2196" containerName="cleanup"
Sep 11 20:30:16.031858 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:16.031744    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="411174cb-1485-4333-ab49-ad3dc2ec2196" containerName="cleanup"
Sep 11 20:30:16.031858 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:16.031762    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b15257a-7c74-4b1d-a88e-4cbe198dd569" containerName="cleanup"
Sep 11 20:30:16.031858 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:16.031769    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b15257a-7c74-4b1d-a88e-4cbe198dd569" containerName="cleanup"
Sep 11 20:30:16.031858 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:16.031818    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b15257a-7c74-4b1d-a88e-4cbe198dd569" containerName="cleanup"
Sep 11 20:30:16.031858 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:16.031826    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="411174cb-1485-4333-ab49-ad3dc2ec2196" containerName="cleanup"
Sep 11 20:30:16.031858 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:16.031832    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcebaf46-aab1-4b87-b80d-d1517576b19a" containerName="cleanup"
Sep 11 20:30:16.035071 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:16.035050    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g"
Sep 11 20:30:16.038210 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:16.038181    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b6453d\""
Sep 11 20:30:16.038435 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:16.038421    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b6453d-dockercfg-pp26b\""
Sep 11 20:30:16.046284 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:16.046261    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g"]
Sep 11 20:30:16.157652 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:16.157614    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811b9aae-8c34-47bd-b30b-8a6d8f957576-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g\" (UID: \"811b9aae-8c34-47bd-b30b-8a6d8f957576\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g"
Sep 11 20:30:16.157842 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:16.157731    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfglq\" (UniqueName: \"kubernetes.io/projected/811b9aae-8c34-47bd-b30b-8a6d8f957576-kube-api-access-pfglq\") pod \"payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g\" (UID: \"811b9aae-8c34-47bd-b30b-8a6d8f957576\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g"
Sep 11 20:30:16.258838 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:16.258800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfglq\" (UniqueName: \"kubernetes.io/projected/811b9aae-8c34-47bd-b30b-8a6d8f957576-kube-api-access-pfglq\") pod \"payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g\" (UID: \"811b9aae-8c34-47bd-b30b-8a6d8f957576\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g"
Sep 11 20:30:16.258981 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:16.258852    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811b9aae-8c34-47bd-b30b-8a6d8f957576-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g\" (UID: \"811b9aae-8c34-47bd-b30b-8a6d8f957576\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g"
Sep 11 20:30:16.259512 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:16.259477    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811b9aae-8c34-47bd-b30b-8a6d8f957576-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g\" (UID: \"811b9aae-8c34-47bd-b30b-8a6d8f957576\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g"
Sep 11 20:30:16.267560 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:16.267537    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfglq\" (UniqueName: \"kubernetes.io/projected/811b9aae-8c34-47bd-b30b-8a6d8f957576-kube-api-access-pfglq\") pod \"payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g\" (UID: \"811b9aae-8c34-47bd-b30b-8a6d8f957576\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g"
Sep 11 20:30:16.347290 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:16.347222    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g"
Sep 11 20:30:16.634007 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:16.633972    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g"]
Sep 11 20:30:16.635964 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:30:16.635930    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811b9aae_8c34_47bd_b30b_8a6d8f957576.slice/crio-cafee661f41919ebe384bc88d594e89824043dd57dd918fa763e9fa0d28a36d1 WatchSource:0}: Error finding container cafee661f41919ebe384bc88d594e89824043dd57dd918fa763e9fa0d28a36d1: Status 404 returned error can't find the container with id cafee661f41919ebe384bc88d594e89824043dd57dd918fa763e9fa0d28a36d1
Sep 11 20:30:17.273908 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:17.273872    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g" event={"ID":"811b9aae-8c34-47bd-b30b-8a6d8f957576","Type":"ContainerStarted","Data":"081cb28fdb93ca6eaaab3e9d8f8052a9bd2fe0acf46f10bc376642a3292d1e8a"}
Sep 11 20:30:17.274397 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:17.273935    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g" event={"ID":"811b9aae-8c34-47bd-b30b-8a6d8f957576","Type":"ContainerStarted","Data":"cafee661f41919ebe384bc88d594e89824043dd57dd918fa763e9fa0d28a36d1"}
Sep 11 20:30:17.274397 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:17.274079    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g"
Sep 11 20:30:17.296780 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:17.296731    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g" podStartSLOduration=1.296716684 podStartE2EDuration="1.296716684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:30:17.294754092 +0000 UTC m=+1787.286916363" watchObservedRunningTime="2026-09-11 20:30:17.296716684 +0000 UTC m=+1787.288878955"
Sep 11 20:30:20.214749 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:20.214710    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:30:28.282177 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:28.282142    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g"
Sep 11 20:30:42.400617 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:42.400586    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jtmt6_76f2ec85-7849-4143-9b55-65fe0133f662/payload-processing/0.log"
Sep 11 20:30:42.521832 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:30:42.521797    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g_811b9aae-8c34-47bd-b30b-8a6d8f957576/payload-processing/0.log"
Sep 11 20:31:00.498691 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.498654    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"]
Sep 11 20:31:00.501924 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.501908    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:31:00.506217 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.506198    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-248ab6\"/\"default-dockercfg-f9jgb\""
Sep 11 20:31:00.506351 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.506274    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-248ab6\"/\"ipp-route-248ab6-kserve-self-signed-certs\""
Sep 11 20:31:00.506351 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.506303    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-248ab6\"/\"openshift-service-ca.crt\""
Sep 11 20:31:00.506535 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.506303    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-248ab6\"/\"kube-root-ca.crt\""
Sep 11 20:31:00.512306 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.512281    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"]
Sep 11 20:31:00.633931 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.633896    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjkbg\" (UniqueName: \"kubernetes.io/projected/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-kube-api-access-bjkbg\") pod \"ipp-route-248ab6-kserve-744c5c96c9-gvxhr\" (UID: \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\") " pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:31:00.634084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.633945    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-home\") pod \"ipp-route-248ab6-kserve-744c5c96c9-gvxhr\" (UID: \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\") " pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:31:00.634084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.633994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-dshm\") pod \"ipp-route-248ab6-kserve-744c5c96c9-gvxhr\" (UID: \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\") " pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:31:00.634084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.634036    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-tmp-dir\") pod \"ipp-route-248ab6-kserve-744c5c96c9-gvxhr\" (UID: \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\") " pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:31:00.634084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.634078    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-tls-certs\") pod \"ipp-route-248ab6-kserve-744c5c96c9-gvxhr\" (UID: \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\") " pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:31:00.634222 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.634107    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-model-cache\") pod \"ipp-route-248ab6-kserve-744c5c96c9-gvxhr\" (UID: \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\") " pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:31:00.735261 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.735225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-home\") pod \"ipp-route-248ab6-kserve-744c5c96c9-gvxhr\" (UID: \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\") " pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:31:00.735420 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.735271    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-dshm\") pod \"ipp-route-248ab6-kserve-744c5c96c9-gvxhr\" (UID: \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\") " pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:31:00.735420 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.735297    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-tmp-dir\") pod \"ipp-route-248ab6-kserve-744c5c96c9-gvxhr\" (UID: \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\") " pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:31:00.735420 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.735316    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-tls-certs\") pod \"ipp-route-248ab6-kserve-744c5c96c9-gvxhr\" (UID: \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\") " pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:31:00.735420 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.735348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-model-cache\") pod \"ipp-route-248ab6-kserve-744c5c96c9-gvxhr\" (UID: \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\") " pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:31:00.735420 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.735375    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjkbg\" (UniqueName: \"kubernetes.io/projected/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-kube-api-access-bjkbg\") pod \"ipp-route-248ab6-kserve-744c5c96c9-gvxhr\" (UID: \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\") " pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:31:00.735758 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.735731    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-tmp-dir\") pod \"ipp-route-248ab6-kserve-744c5c96c9-gvxhr\" (UID: \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\") " pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:31:00.735853 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.735740    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-home\") pod \"ipp-route-248ab6-kserve-744c5c96c9-gvxhr\" (UID: \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\") " pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:31:00.735853 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.735796    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-model-cache\") pod \"ipp-route-248ab6-kserve-744c5c96c9-gvxhr\" (UID: \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\") " pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:31:00.737716 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.737690    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-dshm\") pod \"ipp-route-248ab6-kserve-744c5c96c9-gvxhr\" (UID: \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\") " pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:31:00.738003 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.737984    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-tls-certs\") pod \"ipp-route-248ab6-kserve-744c5c96c9-gvxhr\" (UID: \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\") " pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:31:00.744293 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.744269    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjkbg\" (UniqueName: \"kubernetes.io/projected/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-kube-api-access-bjkbg\") pod \"ipp-route-248ab6-kserve-744c5c96c9-gvxhr\" (UID: \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\") " pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:31:00.813145 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:00.813072    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:31:01.088000 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:01.087969    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"]
Sep 11 20:31:01.089384 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:31:01.089357    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d29a1c1_6954_4d2b_a6fa_6a0176d46638.slice/crio-450ff27e8d38dc105f89f35adba87ddde67b7f1fef299061bda5d26f2877fae0 WatchSource:0}: Error finding container 450ff27e8d38dc105f89f35adba87ddde67b7f1fef299061bda5d26f2877fae0: Status 404 returned error can't find the container with id 450ff27e8d38dc105f89f35adba87ddde67b7f1fef299061bda5d26f2877fae0
Sep 11 20:31:01.519932 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:01.519894    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr" event={"ID":"4d29a1c1-6954-4d2b-a6fa-6a0176d46638","Type":"ContainerStarted","Data":"b3ec1c814300429fdb668327d24fce1b635954dcdfc1cf4921e871cd47602c1a"}
Sep 11 20:31:01.520314 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:01.519940    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr" event={"ID":"4d29a1c1-6954-4d2b-a6fa-6a0176d46638","Type":"ContainerStarted","Data":"450ff27e8d38dc105f89f35adba87ddde67b7f1fef299061bda5d26f2877fae0"}
Sep 11 20:31:01.549741 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:01.549686    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr" podStartSLOduration=1.549671398 podStartE2EDuration="1.549671398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:31:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:31:01.54691031 +0000 UTC m=+1831.539072581" watchObservedRunningTime="2026-09-11 20:31:01.549671398 +0000 UTC m=+1831.541833723"
Sep 11 20:31:04.516622 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:04.516584    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:31:10.813541 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:10.813487    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:31:10.813949 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:10.813558    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:31:10.827206 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:10.827175    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:31:11.585657 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:11.585626    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:31:27.518521 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:27.518473    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:31:33.642713 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:33.642677    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p_20c06bbc-a74c-49b7-aa9e-7d02bb5576c5/payload-processing/0.log"
Sep 11 20:31:33.770844 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:33.770812    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jtmt6_76f2ec85-7849-4143-9b55-65fe0133f662/payload-processing/0.log"
Sep 11 20:31:33.889870 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:33.889846    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g_811b9aae-8c34-47bd-b30b-8a6d8f957576/payload-processing/0.log"
Sep 11 20:31:41.482123 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:41.482088    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67"]
Sep 11 20:31:41.485693 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:41.485674    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67"
Sep 11 20:31:41.488508 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:41.488476    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-5de5ea\""
Sep 11 20:31:41.497152 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:41.497123    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67"]
Sep 11 20:31:41.589152 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:41.589118    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd58r\" (UniqueName: \"kubernetes.io/projected/39784825-b198-4f72-a420-20f4eeeec523-kube-api-access-sd58r\") pod \"payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67\" (UID: \"39784825-b198-4f72-a420-20f4eeeec523\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67"
Sep 11 20:31:41.589314 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:41.589220    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39784825-b198-4f72-a420-20f4eeeec523-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67\" (UID: \"39784825-b198-4f72-a420-20f4eeeec523\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67"
Sep 11 20:31:41.690584 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:41.690553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39784825-b198-4f72-a420-20f4eeeec523-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67\" (UID: \"39784825-b198-4f72-a420-20f4eeeec523\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67"
Sep 11 20:31:41.690747 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:41.690593    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd58r\" (UniqueName: \"kubernetes.io/projected/39784825-b198-4f72-a420-20f4eeeec523-kube-api-access-sd58r\") pod \"payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67\" (UID: \"39784825-b198-4f72-a420-20f4eeeec523\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67"
Sep 11 20:31:41.691221 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:41.691202    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39784825-b198-4f72-a420-20f4eeeec523-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67\" (UID: \"39784825-b198-4f72-a420-20f4eeeec523\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67"
Sep 11 20:31:41.705552 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:41.698519    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd58r\" (UniqueName: \"kubernetes.io/projected/39784825-b198-4f72-a420-20f4eeeec523-kube-api-access-sd58r\") pod \"payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67\" (UID: \"39784825-b198-4f72-a420-20f4eeeec523\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67"
Sep 11 20:31:41.797285 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:41.797198    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67"
Sep 11 20:31:42.068618 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:42.068592    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67"]
Sep 11 20:31:42.070882 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:31:42.070857    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39784825_b198_4f72_a420_20f4eeeec523.slice/crio-07d5eca9a8b15ea391b5b19563d15099aa83ad10e27144e247c9a02a464afac8 WatchSource:0}: Error finding container 07d5eca9a8b15ea391b5b19563d15099aa83ad10e27144e247c9a02a464afac8: Status 404 returned error can't find the container with id 07d5eca9a8b15ea391b5b19563d15099aa83ad10e27144e247c9a02a464afac8
Sep 11 20:31:42.748255 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:42.748205    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67" event={"ID":"39784825-b198-4f72-a420-20f4eeeec523","Type":"ContainerStarted","Data":"35f42f4278e3526776e3763d11ace7494a5c413360fbef6d45d50dc82a5eedd4"}
Sep 11 20:31:42.748676 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:42.748276    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67"
Sep 11 20:31:42.748676 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:42.748295    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67" event={"ID":"39784825-b198-4f72-a420-20f4eeeec523","Type":"ContainerStarted","Data":"07d5eca9a8b15ea391b5b19563d15099aa83ad10e27144e247c9a02a464afac8"}
Sep 11 20:31:42.765627 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:42.765582    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67" podStartSLOduration=1.7655672089999999 podStartE2EDuration="1.765567209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:31:42.763941448 +0000 UTC m=+1872.756103717" watchObservedRunningTime="2026-09-11 20:31:42.765567209 +0000 UTC m=+1872.757729478"
Sep 11 20:31:46.312676 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:46.312643    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:31:53.757069 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:31:53.756996    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67"
Sep 11 20:32:08.162583 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:08.162545    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5de5ea-bmr6m"]
Sep 11 20:32:08.166079 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:08.166062    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5de5ea-bmr6m"
Sep 11 20:32:08.168487 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:08.168467    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:32:08.168620 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:08.168528    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7dbs5\""
Sep 11 20:32:08.173239 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:08.173214    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5de5ea-bmr6m"]
Sep 11 20:32:08.214692 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:08.214655    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d9a234-e15c-446b-899c-733283add014-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5de5ea-bmr6m\" (UID: \"a4d9a234-e15c-446b-899c-733283add014\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5de5ea-bmr6m"
Sep 11 20:32:08.315377 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:08.315344    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d9a234-e15c-446b-899c-733283add014-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5de5ea-bmr6m\" (UID: \"a4d9a234-e15c-446b-899c-733283add014\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5de5ea-bmr6m"
Sep 11 20:32:08.315986 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:08.315965    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d9a234-e15c-446b-899c-733283add014-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5de5ea-bmr6m\" (UID: \"a4d9a234-e15c-446b-899c-733283add014\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5de5ea-bmr6m"
Sep 11 20:32:08.478229 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:08.478200    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5de5ea-bmr6m"
Sep 11 20:32:08.752192 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:08.752164    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5de5ea-bmr6m"]
Sep 11 20:32:08.753992 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:32:08.753960    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d9a234_e15c_446b_899c_733283add014.slice/crio-dfe3c67d1a48c685cbf114d8b201981b28b9bacea19fae03a036e34b8feaffa2 WatchSource:0}: Error finding container dfe3c67d1a48c685cbf114d8b201981b28b9bacea19fae03a036e34b8feaffa2: Status 404 returned error can't find the container with id dfe3c67d1a48c685cbf114d8b201981b28b9bacea19fae03a036e34b8feaffa2
Sep 11 20:32:08.900478 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:08.900440    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5de5ea-bmr6m" event={"ID":"a4d9a234-e15c-446b-899c-733283add014","Type":"ContainerStarted","Data":"dfe3c67d1a48c685cbf114d8b201981b28b9bacea19fae03a036e34b8feaffa2"}
Sep 11 20:32:09.908521 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:09.908468    2580 generic.go:356] "Generic (PLEG): container finished" podID="a4d9a234-e15c-446b-899c-733283add014" containerID="f1f394ad9624746c91f4020c2d4c79471a175af53620fe1f4df01d338ed2e1e6" exitCode=0
Sep 11 20:32:09.908918 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:09.908566    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5de5ea-bmr6m" event={"ID":"a4d9a234-e15c-446b-899c-733283add014","Type":"ContainerDied","Data":"f1f394ad9624746c91f4020c2d4c79471a175af53620fe1f4df01d338ed2e1e6"}
Sep 11 20:32:11.049973 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:11.049948    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5de5ea-bmr6m"
Sep 11 20:32:11.139943 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:11.139911    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d9a234-e15c-446b-899c-733283add014-maas-api-service-ca\") pod \"a4d9a234-e15c-446b-899c-733283add014\" (UID: \"a4d9a234-e15c-446b-899c-733283add014\") "
Sep 11 20:32:11.140257 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:11.140227    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4d9a234-e15c-446b-899c-733283add014-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a4d9a234-e15c-446b-899c-733283add014" (UID: "a4d9a234-e15c-446b-899c-733283add014"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:32:11.241309 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:11.241276    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d9a234-e15c-446b-899c-733283add014-maas-api-service-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:32:11.920434 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:11.920401    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5de5ea-bmr6m"
Sep 11 20:32:11.920659 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:11.920450    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5de5ea-bmr6m" event={"ID":"a4d9a234-e15c-446b-899c-733283add014","Type":"ContainerDied","Data":"dfe3c67d1a48c685cbf114d8b201981b28b9bacea19fae03a036e34b8feaffa2"}
Sep 11 20:32:11.920659 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:11.920518    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfe3c67d1a48c685cbf114d8b201981b28b9bacea19fae03a036e34b8feaffa2"
Sep 11 20:32:19.250564 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:19.250528    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67"]
Sep 11 20:32:19.251110 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:19.250961    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67" podUID="39784825-b198-4f72-a420-20f4eeeec523" containerName="payload-pre-processing" containerID="cri-o://35f42f4278e3526776e3763d11ace7494a5c413360fbef6d45d50dc82a5eedd4" gracePeriod=30
Sep 11 20:32:19.403773 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:19.403734    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5de5ea-bmr6m"]
Sep 11 20:32:19.406421 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:19.406396    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5de5ea-bmr6m"]
Sep 11 20:32:19.498449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:19.498419    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67"
Sep 11 20:32:19.614974 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:19.614890    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd58r\" (UniqueName: \"kubernetes.io/projected/39784825-b198-4f72-a420-20f4eeeec523-kube-api-access-sd58r\") pod \"39784825-b198-4f72-a420-20f4eeeec523\" (UID: \"39784825-b198-4f72-a420-20f4eeeec523\") "
Sep 11 20:32:19.614974 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:19.614946    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39784825-b198-4f72-a420-20f4eeeec523-plugins-config-volume\") pod \"39784825-b198-4f72-a420-20f4eeeec523\" (UID: \"39784825-b198-4f72-a420-20f4eeeec523\") "
Sep 11 20:32:19.615352 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:19.615330    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39784825-b198-4f72-a420-20f4eeeec523-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "39784825-b198-4f72-a420-20f4eeeec523" (UID: "39784825-b198-4f72-a420-20f4eeeec523"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:32:19.616989 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:19.616971    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39784825-b198-4f72-a420-20f4eeeec523-kube-api-access-sd58r" (OuterVolumeSpecName: "kube-api-access-sd58r") pod "39784825-b198-4f72-a420-20f4eeeec523" (UID: "39784825-b198-4f72-a420-20f4eeeec523"). InnerVolumeSpecName "kube-api-access-sd58r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:32:19.715843 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:19.715812    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39784825-b198-4f72-a420-20f4eeeec523-plugins-config-volume\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:32:19.715843 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:19.715840    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd58r\" (UniqueName: \"kubernetes.io/projected/39784825-b198-4f72-a420-20f4eeeec523-kube-api-access-sd58r\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:32:19.968827 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:19.968798    2580 generic.go:356] "Generic (PLEG): container finished" podID="39784825-b198-4f72-a420-20f4eeeec523" containerID="35f42f4278e3526776e3763d11ace7494a5c413360fbef6d45d50dc82a5eedd4" exitCode=0
Sep 11 20:32:19.968999 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:19.968842    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67"
Sep 11 20:32:19.968999 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:19.968890    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67" event={"ID":"39784825-b198-4f72-a420-20f4eeeec523","Type":"ContainerDied","Data":"35f42f4278e3526776e3763d11ace7494a5c413360fbef6d45d50dc82a5eedd4"}
Sep 11 20:32:19.968999 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:19.968942    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67" event={"ID":"39784825-b198-4f72-a420-20f4eeeec523","Type":"ContainerDied","Data":"07d5eca9a8b15ea391b5b19563d15099aa83ad10e27144e247c9a02a464afac8"}
Sep 11 20:32:19.968999 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:19.968966    2580 scope.go:117] "RemoveContainer" containerID="35f42f4278e3526776e3763d11ace7494a5c413360fbef6d45d50dc82a5eedd4"
Sep 11 20:32:19.978850 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:19.978832    2580 scope.go:117] "RemoveContainer" containerID="35f42f4278e3526776e3763d11ace7494a5c413360fbef6d45d50dc82a5eedd4"
Sep 11 20:32:19.979123 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:32:19.979104    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35f42f4278e3526776e3763d11ace7494a5c413360fbef6d45d50dc82a5eedd4\": container with ID starting with 35f42f4278e3526776e3763d11ace7494a5c413360fbef6d45d50dc82a5eedd4 not found: ID does not exist" containerID="35f42f4278e3526776e3763d11ace7494a5c413360fbef6d45d50dc82a5eedd4"
Sep 11 20:32:19.979193 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:19.979134    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f42f4278e3526776e3763d11ace7494a5c413360fbef6d45d50dc82a5eedd4"} err="failed to get container status \"35f42f4278e3526776e3763d11ace7494a5c413360fbef6d45d50dc82a5eedd4\": rpc error: code = NotFound desc = could not find container \"35f42f4278e3526776e3763d11ace7494a5c413360fbef6d45d50dc82a5eedd4\": container with ID starting with 35f42f4278e3526776e3763d11ace7494a5c413360fbef6d45d50dc82a5eedd4 not found: ID does not exist"
Sep 11 20:32:19.989603 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:19.989581    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67"]
Sep 11 20:32:19.994209 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:19.994188    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5de5ea-76b4475f6b-8gf67"]
Sep 11 20:32:20.440500 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:20.440457    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39784825-b198-4f72-a420-20f4eeeec523" path="/var/lib/kubelet/pods/39784825-b198-4f72-a420-20f4eeeec523/volumes"
Sep 11 20:32:20.440857 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:20.440808    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4d9a234-e15c-446b-899c-733283add014" path="/var/lib/kubelet/pods/a4d9a234-e15c-446b-899c-733283add014/volumes"
Sep 11 20:32:23.998990 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:23.998955    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:32:29.731256 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:29.731217    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-248ab6-kcwvh"]
Sep 11 20:32:29.731746 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:29.731623    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4d9a234-e15c-446b-899c-733283add014" containerName="revoke-keys"
Sep 11 20:32:29.731746 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:29.731636    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4d9a234-e15c-446b-899c-733283add014" containerName="revoke-keys"
Sep 11 20:32:29.731746 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:29.731649    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39784825-b198-4f72-a420-20f4eeeec523" containerName="payload-pre-processing"
Sep 11 20:32:29.731746 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:29.731656    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="39784825-b198-4f72-a420-20f4eeeec523" containerName="payload-pre-processing"
Sep 11 20:32:29.731746 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:29.731712    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4d9a234-e15c-446b-899c-733283add014" containerName="revoke-keys"
Sep 11 20:32:29.731746 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:29.731721    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="39784825-b198-4f72-a420-20f4eeeec523" containerName="payload-pre-processing"
Sep 11 20:32:29.736149 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:29.736132    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-248ab6-kcwvh"
Sep 11 20:32:29.739756 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:29.739728    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:32:29.739942 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:29.739727    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7dbs5\""
Sep 11 20:32:29.742960 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:29.742938    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-248ab6-kcwvh"]
Sep 11 20:32:29.807552 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:29.807522    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/809a9e98-d28e-4293-9d9e-1735b10262ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-248ab6-kcwvh\" (UID: \"809a9e98-d28e-4293-9d9e-1735b10262ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-248ab6-kcwvh"
Sep 11 20:32:29.908950 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:29.908914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/809a9e98-d28e-4293-9d9e-1735b10262ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-248ab6-kcwvh\" (UID: \"809a9e98-d28e-4293-9d9e-1735b10262ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-248ab6-kcwvh"
Sep 11 20:32:29.909555 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:29.909536    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/809a9e98-d28e-4293-9d9e-1735b10262ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-248ab6-kcwvh\" (UID: \"809a9e98-d28e-4293-9d9e-1735b10262ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-248ab6-kcwvh"
Sep 11 20:32:30.049109 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:30.049025    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-248ab6-kcwvh"
Sep 11 20:32:30.319579 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:30.319553    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-248ab6-kcwvh"]
Sep 11 20:32:30.321266 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:32:30.321241    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809a9e98_d28e_4293_9d9e_1735b10262ae.slice/crio-a000312c0ba887d10393d452d15d22358175dcfbf1c92271d81d5cc88322d79c WatchSource:0}: Error finding container a000312c0ba887d10393d452d15d22358175dcfbf1c92271d81d5cc88322d79c: Status 404 returned error can't find the container with id a000312c0ba887d10393d452d15d22358175dcfbf1c92271d81d5cc88322d79c
Sep 11 20:32:30.960568 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:30.960540    2580 scope.go:117] "RemoveContainer" containerID="d8a67bc3b2c48bf69eb3305c76597fa2bb0717fa7e1a04e7e6a928556abfbe0e"
Sep 11 20:32:30.969257 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:30.969238    2580 scope.go:117] "RemoveContainer" containerID="b9ee65038617b4117506f489c4e682e583b3b6a17f2fac2370d96f17e8415def"
Sep 11 20:32:30.977603 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:30.977584    2580 scope.go:117] "RemoveContainer" containerID="814c05b786782253bbae55729a0de152e081c1ea92ec30006d6b62f466d7adb8"
Sep 11 20:32:31.030881 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:31.030851    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-248ab6-kcwvh" event={"ID":"809a9e98-d28e-4293-9d9e-1735b10262ae","Type":"ContainerStarted","Data":"4b93f2b87d02916cb53c843c88dddca39e6f6bd6cb806c3722c56248dc21b229"}
Sep 11 20:32:31.030979 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:31.030898    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-248ab6-kcwvh" event={"ID":"809a9e98-d28e-4293-9d9e-1735b10262ae","Type":"ContainerStarted","Data":"a000312c0ba887d10393d452d15d22358175dcfbf1c92271d81d5cc88322d79c"}
Sep 11 20:32:31.045304 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:31.045264    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-248ab6-kcwvh" podStartSLOduration=2.045250329 podStartE2EDuration="2.045250329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:32:31.044133008 +0000 UTC m=+1921.036295279" watchObservedRunningTime="2026-09-11 20:32:31.045250329 +0000 UTC m=+1921.037412599"
Sep 11 20:32:32.037046 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:32.037006    2580 generic.go:356] "Generic (PLEG): container finished" podID="809a9e98-d28e-4293-9d9e-1735b10262ae" containerID="4b93f2b87d02916cb53c843c88dddca39e6f6bd6cb806c3722c56248dc21b229" exitCode=0
Sep 11 20:32:32.037467 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:32.037095    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-248ab6-kcwvh" event={"ID":"809a9e98-d28e-4293-9d9e-1735b10262ae","Type":"ContainerDied","Data":"4b93f2b87d02916cb53c843c88dddca39e6f6bd6cb806c3722c56248dc21b229"}
Sep 11 20:32:33.184594 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:33.184568    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-248ab6-kcwvh"
Sep 11 20:32:33.340107 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:33.340027    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/809a9e98-d28e-4293-9d9e-1735b10262ae-maas-api-service-ca\") pod \"809a9e98-d28e-4293-9d9e-1735b10262ae\" (UID: \"809a9e98-d28e-4293-9d9e-1735b10262ae\") "
Sep 11 20:32:33.340364 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:33.340343    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/809a9e98-d28e-4293-9d9e-1735b10262ae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "809a9e98-d28e-4293-9d9e-1735b10262ae" (UID: "809a9e98-d28e-4293-9d9e-1735b10262ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:32:33.440479 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:33.440448    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/809a9e98-d28e-4293-9d9e-1735b10262ae-maas-api-service-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:32:34.049068 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:34.049041    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-248ab6-kcwvh"
Sep 11 20:32:34.049247 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:34.049040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-248ab6-kcwvh" event={"ID":"809a9e98-d28e-4293-9d9e-1735b10262ae","Type":"ContainerDied","Data":"a000312c0ba887d10393d452d15d22358175dcfbf1c92271d81d5cc88322d79c"}
Sep 11 20:32:34.049247 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:34.049171    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a000312c0ba887d10393d452d15d22358175dcfbf1c92271d81d5cc88322d79c"
Sep 11 20:32:43.000096 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:43.000051    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:32:45.336726 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:45.336684    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p"]
Sep 11 20:32:45.337181 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:45.337033    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p" podUID="20c06bbc-a74c-49b7-aa9e-7d02bb5576c5" containerName="payload-processing" containerID="cri-o://66d070a22eb7f6837fd66555fb4a87fce3377e12f6980542083b45034b836bbf" gracePeriod=30
Sep 11 20:32:45.498395 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:45.498361    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-248ab6-kcwvh"]
Sep 11 20:32:45.504805 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:45.504739    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-248ab6-kcwvh"]
Sep 11 20:32:45.585632 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:45.585603    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p"
Sep 11 20:32:45.642936 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:45.642865    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c06bbc-a74c-49b7-aa9e-7d02bb5576c5-plugins-config-volume\") pod \"20c06bbc-a74c-49b7-aa9e-7d02bb5576c5\" (UID: \"20c06bbc-a74c-49b7-aa9e-7d02bb5576c5\") "
Sep 11 20:32:45.642936 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:45.642904    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmdnl\" (UniqueName: \"kubernetes.io/projected/20c06bbc-a74c-49b7-aa9e-7d02bb5576c5-kube-api-access-kmdnl\") pod \"20c06bbc-a74c-49b7-aa9e-7d02bb5576c5\" (UID: \"20c06bbc-a74c-49b7-aa9e-7d02bb5576c5\") "
Sep 11 20:32:45.643227 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:45.643202    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c06bbc-a74c-49b7-aa9e-7d02bb5576c5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "20c06bbc-a74c-49b7-aa9e-7d02bb5576c5" (UID: "20c06bbc-a74c-49b7-aa9e-7d02bb5576c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:32:45.645021 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:45.644997    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c06bbc-a74c-49b7-aa9e-7d02bb5576c5-kube-api-access-kmdnl" (OuterVolumeSpecName: "kube-api-access-kmdnl") pod "20c06bbc-a74c-49b7-aa9e-7d02bb5576c5" (UID: "20c06bbc-a74c-49b7-aa9e-7d02bb5576c5"). InnerVolumeSpecName "kube-api-access-kmdnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:32:45.744473 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:45.744440    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c06bbc-a74c-49b7-aa9e-7d02bb5576c5-plugins-config-volume\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:32:45.744473 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:45.744468    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmdnl\" (UniqueName: \"kubernetes.io/projected/20c06bbc-a74c-49b7-aa9e-7d02bb5576c5-kube-api-access-kmdnl\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:32:46.116340 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:46.116301    2580 generic.go:356] "Generic (PLEG): container finished" podID="20c06bbc-a74c-49b7-aa9e-7d02bb5576c5" containerID="66d070a22eb7f6837fd66555fb4a87fce3377e12f6980542083b45034b836bbf" exitCode=0
Sep 11 20:32:46.116534 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:46.116367    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p"
Sep 11 20:32:46.116534 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:46.116383    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p" event={"ID":"20c06bbc-a74c-49b7-aa9e-7d02bb5576c5","Type":"ContainerDied","Data":"66d070a22eb7f6837fd66555fb4a87fce3377e12f6980542083b45034b836bbf"}
Sep 11 20:32:46.116534 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:46.116462    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p" event={"ID":"20c06bbc-a74c-49b7-aa9e-7d02bb5576c5","Type":"ContainerDied","Data":"3f4635c140ba199640772292445224e6144a1ac2f3caad80df32d5125540a2f1"}
Sep 11 20:32:46.116534 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:46.116486    2580 scope.go:117] "RemoveContainer" containerID="66d070a22eb7f6837fd66555fb4a87fce3377e12f6980542083b45034b836bbf"
Sep 11 20:32:46.126439 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:46.126421    2580 scope.go:117] "RemoveContainer" containerID="66d070a22eb7f6837fd66555fb4a87fce3377e12f6980542083b45034b836bbf"
Sep 11 20:32:46.126741 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:32:46.126724    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d070a22eb7f6837fd66555fb4a87fce3377e12f6980542083b45034b836bbf\": container with ID starting with 66d070a22eb7f6837fd66555fb4a87fce3377e12f6980542083b45034b836bbf not found: ID does not exist" containerID="66d070a22eb7f6837fd66555fb4a87fce3377e12f6980542083b45034b836bbf"
Sep 11 20:32:46.126813 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:46.126751    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d070a22eb7f6837fd66555fb4a87fce3377e12f6980542083b45034b836bbf"} err="failed to get container status \"66d070a22eb7f6837fd66555fb4a87fce3377e12f6980542083b45034b836bbf\": rpc error: code = NotFound desc = could not find container \"66d070a22eb7f6837fd66555fb4a87fce3377e12f6980542083b45034b836bbf\": container with ID starting with 66d070a22eb7f6837fd66555fb4a87fce3377e12f6980542083b45034b836bbf not found: ID does not exist"
Sep 11 20:32:46.137959 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:46.137936    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p"]
Sep 11 20:32:46.141345 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:46.141324    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-248ab6-6557dcbd9f-gjq5p"]
Sep 11 20:32:46.440769 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:46.440736    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20c06bbc-a74c-49b7-aa9e-7d02bb5576c5" path="/var/lib/kubelet/pods/20c06bbc-a74c-49b7-aa9e-7d02bb5576c5/volumes"
Sep 11 20:32:46.441121 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:46.441063    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="809a9e98-d28e-4293-9d9e-1735b10262ae" path="/var/lib/kubelet/pods/809a9e98-d28e-4293-9d9e-1735b10262ae/volumes"
Sep 11 20:32:50.842711 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:50.842680    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"]
Sep 11 20:32:55.400917 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:32:55.400883    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:33:05.861639 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:05.861574    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr" podUID="4d29a1c1-6954-4d2b-a6fa-6a0176d46638" containerName="main" containerID="cri-o://b3ec1c814300429fdb668327d24fce1b635954dcdfc1cf4921e871cd47602c1a" gracePeriod=45
Sep 11 20:33:06.113384 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.113321    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:33:06.222823 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.222797    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjkbg\" (UniqueName: \"kubernetes.io/projected/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-kube-api-access-bjkbg\") pod \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\" (UID: \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\") "
Sep 11 20:33:06.223012 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.222862    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-tmp-dir\") pod \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\" (UID: \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\") "
Sep 11 20:33:06.223012 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.222888    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-model-cache\") pod \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\" (UID: \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\") "
Sep 11 20:33:06.223012 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.222915    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-home\") pod \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\" (UID: \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\") "
Sep 11 20:33:06.223012 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.222943    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-tls-certs\") pod \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\" (UID: \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\") "
Sep 11 20:33:06.223012 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.222972    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-dshm\") pod \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\" (UID: \"4d29a1c1-6954-4d2b-a6fa-6a0176d46638\") "
Sep 11 20:33:06.223267 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.223114    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d29a1c1-6954-4d2b-a6fa-6a0176d46638" (UID: "4d29a1c1-6954-4d2b-a6fa-6a0176d46638"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:33:06.223267 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.223158    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d29a1c1-6954-4d2b-a6fa-6a0176d46638" (UID: "4d29a1c1-6954-4d2b-a6fa-6a0176d46638"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:33:06.223267 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.223237    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-home" (OuterVolumeSpecName: "home") pod "4d29a1c1-6954-4d2b-a6fa-6a0176d46638" (UID: "4d29a1c1-6954-4d2b-a6fa-6a0176d46638"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:33:06.223411 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.223381    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-tmp-dir\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:33:06.223411 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.223396    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-model-cache\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:33:06.223411 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.223405    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-home\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:33:06.225210 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.225182    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-kube-api-access-bjkbg" (OuterVolumeSpecName: "kube-api-access-bjkbg") pod "4d29a1c1-6954-4d2b-a6fa-6a0176d46638" (UID: "4d29a1c1-6954-4d2b-a6fa-6a0176d46638"). InnerVolumeSpecName "kube-api-access-bjkbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:33:06.225342 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.225202    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d29a1c1-6954-4d2b-a6fa-6a0176d46638" (UID: "4d29a1c1-6954-4d2b-a6fa-6a0176d46638"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:33:06.225342 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.225291    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-dshm" (OuterVolumeSpecName: "dshm") pod "4d29a1c1-6954-4d2b-a6fa-6a0176d46638" (UID: "4d29a1c1-6954-4d2b-a6fa-6a0176d46638"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:33:06.231168 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.231142    2580 generic.go:356] "Generic (PLEG): container finished" podID="4d29a1c1-6954-4d2b-a6fa-6a0176d46638" containerID="b3ec1c814300429fdb668327d24fce1b635954dcdfc1cf4921e871cd47602c1a" exitCode=0
Sep 11 20:33:06.231294 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.231185    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"
Sep 11 20:33:06.231294 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.231219    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr" event={"ID":"4d29a1c1-6954-4d2b-a6fa-6a0176d46638","Type":"ContainerDied","Data":"b3ec1c814300429fdb668327d24fce1b635954dcdfc1cf4921e871cd47602c1a"}
Sep 11 20:33:06.231294 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.231263    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr" event={"ID":"4d29a1c1-6954-4d2b-a6fa-6a0176d46638","Type":"ContainerDied","Data":"450ff27e8d38dc105f89f35adba87ddde67b7f1fef299061bda5d26f2877fae0"}
Sep 11 20:33:06.231294 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.231281    2580 scope.go:117] "RemoveContainer" containerID="b3ec1c814300429fdb668327d24fce1b635954dcdfc1cf4921e871cd47602c1a"
Sep 11 20:33:06.244265 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.244249    2580 scope.go:117] "RemoveContainer" containerID="b3ec1c814300429fdb668327d24fce1b635954dcdfc1cf4921e871cd47602c1a"
Sep 11 20:33:06.244563 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:33:06.244543    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3ec1c814300429fdb668327d24fce1b635954dcdfc1cf4921e871cd47602c1a\": container with ID starting with b3ec1c814300429fdb668327d24fce1b635954dcdfc1cf4921e871cd47602c1a not found: ID does not exist" containerID="b3ec1c814300429fdb668327d24fce1b635954dcdfc1cf4921e871cd47602c1a"
Sep 11 20:33:06.244657 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.244569    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3ec1c814300429fdb668327d24fce1b635954dcdfc1cf4921e871cd47602c1a"} err="failed to get container status \"b3ec1c814300429fdb668327d24fce1b635954dcdfc1cf4921e871cd47602c1a\": rpc error: code = NotFound desc = could not find container \"b3ec1c814300429fdb668327d24fce1b635954dcdfc1cf4921e871cd47602c1a\": container with ID starting with b3ec1c814300429fdb668327d24fce1b635954dcdfc1cf4921e871cd47602c1a not found: ID does not exist"
Sep 11 20:33:06.253871 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.253847    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"]
Sep 11 20:33:06.257665 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.257645    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-248ab6/ipp-route-248ab6-kserve-744c5c96c9-gvxhr"]
Sep 11 20:33:06.324148 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.324120    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjkbg\" (UniqueName: \"kubernetes.io/projected/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-kube-api-access-bjkbg\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:33:06.324148 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.324143    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-tls-certs\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:33:06.324148 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.324154    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d29a1c1-6954-4d2b-a6fa-6a0176d46638-dshm\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:33:06.441779 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:06.441750    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d29a1c1-6954-4d2b-a6fa-6a0176d46638" path="/var/lib/kubelet/pods/4d29a1c1-6954-4d2b-a6fa-6a0176d46638/volumes"
Sep 11 20:33:31.008466 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:31.008433    2580 scope.go:117] "RemoveContainer" containerID="9a0b5de23c7ae27930acdc58209cd508f1812b8039eaff6377f0283ef6f87c4e"
Sep 11 20:33:31.016854 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:31.016831    2580 scope.go:117] "RemoveContainer" containerID="ca4dff15da2cdc7695e0321c57d21072cb22ea7285434a3271d77510bb5c52ba"
Sep 11 20:33:33.934661 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:33.934624    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6453d-8rrcz"]
Sep 11 20:33:33.935021 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:33.934962    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="809a9e98-d28e-4293-9d9e-1735b10262ae" containerName="revoke-keys"
Sep 11 20:33:33.935021 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:33.934973    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="809a9e98-d28e-4293-9d9e-1735b10262ae" containerName="revoke-keys"
Sep 11 20:33:33.935021 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:33.934983    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d29a1c1-6954-4d2b-a6fa-6a0176d46638" containerName="main"
Sep 11 20:33:33.935021 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:33.934988    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d29a1c1-6954-4d2b-a6fa-6a0176d46638" containerName="main"
Sep 11 20:33:33.935021 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:33.935000    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20c06bbc-a74c-49b7-aa9e-7d02bb5576c5" containerName="payload-processing"
Sep 11 20:33:33.935021 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:33.935007    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="20c06bbc-a74c-49b7-aa9e-7d02bb5576c5" containerName="payload-processing"
Sep 11 20:33:33.935214 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:33.935068    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d29a1c1-6954-4d2b-a6fa-6a0176d46638" containerName="main"
Sep 11 20:33:33.935214 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:33.935077    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="809a9e98-d28e-4293-9d9e-1735b10262ae" containerName="revoke-keys"
Sep 11 20:33:33.935214 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:33.935084    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="20c06bbc-a74c-49b7-aa9e-7d02bb5576c5" containerName="payload-processing"
Sep 11 20:33:33.939591 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:33.939572    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6453d-8rrcz"
Sep 11 20:33:33.943099 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:33.943072    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:33:33.943289 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:33.943271    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7dbs5\""
Sep 11 20:33:33.945574 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:33.945553    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6453d-8rrcz"]
Sep 11 20:33:33.961080 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:33.961056    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4ddfd2-7320-40ae-ab61-9c7d86489c63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b6453d-8rrcz\" (UID: \"6c4ddfd2-7320-40ae-ab61-9c7d86489c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6453d-8rrcz"
Sep 11 20:33:34.061969 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:34.061936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4ddfd2-7320-40ae-ab61-9c7d86489c63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b6453d-8rrcz\" (UID: \"6c4ddfd2-7320-40ae-ab61-9c7d86489c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6453d-8rrcz"
Sep 11 20:33:34.062594 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:34.062567    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4ddfd2-7320-40ae-ab61-9c7d86489c63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b6453d-8rrcz\" (UID: \"6c4ddfd2-7320-40ae-ab61-9c7d86489c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6453d-8rrcz"
Sep 11 20:33:34.251013 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:34.250932    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6453d-8rrcz"
Sep 11 20:33:34.520787 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:34.520761    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6453d-8rrcz"]
Sep 11 20:33:34.522195 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:33:34.522162    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c4ddfd2_7320_40ae_ab61_9c7d86489c63.slice/crio-e99bffa76b3fc685afc6086e3c44559a17b697c47ba1b2b2c158f9a4197b62db WatchSource:0}: Error finding container e99bffa76b3fc685afc6086e3c44559a17b697c47ba1b2b2c158f9a4197b62db: Status 404 returned error can't find the container with id e99bffa76b3fc685afc6086e3c44559a17b697c47ba1b2b2c158f9a4197b62db
Sep 11 20:33:35.387415 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:35.387382    2580 generic.go:356] "Generic (PLEG): container finished" podID="6c4ddfd2-7320-40ae-ab61-9c7d86489c63" containerID="2df2acc8f65b40631169671e64179e3bc89e4d4dea689b262c64b6f1081aada6" exitCode=0
Sep 11 20:33:35.387822 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:35.387430    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6453d-8rrcz" event={"ID":"6c4ddfd2-7320-40ae-ab61-9c7d86489c63","Type":"ContainerDied","Data":"2df2acc8f65b40631169671e64179e3bc89e4d4dea689b262c64b6f1081aada6"}
Sep 11 20:33:35.387822 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:35.387487    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6453d-8rrcz" event={"ID":"6c4ddfd2-7320-40ae-ab61-9c7d86489c63","Type":"ContainerStarted","Data":"e99bffa76b3fc685afc6086e3c44559a17b697c47ba1b2b2c158f9a4197b62db"}
Sep 11 20:33:36.519846 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:36.519822    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6453d-8rrcz"
Sep 11 20:33:36.578185 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:36.578151    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4ddfd2-7320-40ae-ab61-9c7d86489c63-maas-api-service-ca\") pod \"6c4ddfd2-7320-40ae-ab61-9c7d86489c63\" (UID: \"6c4ddfd2-7320-40ae-ab61-9c7d86489c63\") "
Sep 11 20:33:36.578543 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:36.578517    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c4ddfd2-7320-40ae-ab61-9c7d86489c63-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6c4ddfd2-7320-40ae-ab61-9c7d86489c63" (UID: "6c4ddfd2-7320-40ae-ab61-9c7d86489c63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:33:36.678809 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:36.678728    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4ddfd2-7320-40ae-ab61-9c7d86489c63-maas-api-service-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:33:37.399559 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:37.399518    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6453d-8rrcz" event={"ID":"6c4ddfd2-7320-40ae-ab61-9c7d86489c63","Type":"ContainerDied","Data":"e99bffa76b3fc685afc6086e3c44559a17b697c47ba1b2b2c158f9a4197b62db"}
Sep 11 20:33:37.399816 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:37.399608    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e99bffa76b3fc685afc6086e3c44559a17b697c47ba1b2b2c158f9a4197b62db"
Sep 11 20:33:37.399816 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:37.399527    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6453d-8rrcz"
Sep 11 20:33:44.447519 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:44.443561    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g"]
Sep 11 20:33:44.447519 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:44.445255    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g" podUID="811b9aae-8c34-47bd-b30b-8a6d8f957576" containerName="payload-processing" containerID="cri-o://081cb28fdb93ca6eaaab3e9d8f8052a9bd2fe0acf46f10bc376642a3292d1e8a" gracePeriod=30
Sep 11 20:33:44.607116 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:44.607061    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6453d-8rrcz"]
Sep 11 20:33:44.609993 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:44.609966    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b6453d-8rrcz"]
Sep 11 20:33:44.712552 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:44.712519    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g"
Sep 11 20:33:44.732287 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:44.732256    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811b9aae-8c34-47bd-b30b-8a6d8f957576-plugins-config-volume\") pod \"811b9aae-8c34-47bd-b30b-8a6d8f957576\" (UID: \"811b9aae-8c34-47bd-b30b-8a6d8f957576\") "
Sep 11 20:33:44.732428 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:44.732305    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfglq\" (UniqueName: \"kubernetes.io/projected/811b9aae-8c34-47bd-b30b-8a6d8f957576-kube-api-access-pfglq\") pod \"811b9aae-8c34-47bd-b30b-8a6d8f957576\" (UID: \"811b9aae-8c34-47bd-b30b-8a6d8f957576\") "
Sep 11 20:33:44.732684 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:44.732653    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/811b9aae-8c34-47bd-b30b-8a6d8f957576-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "811b9aae-8c34-47bd-b30b-8a6d8f957576" (UID: "811b9aae-8c34-47bd-b30b-8a6d8f957576"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:33:44.734916 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:44.734888    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811b9aae-8c34-47bd-b30b-8a6d8f957576-kube-api-access-pfglq" (OuterVolumeSpecName: "kube-api-access-pfglq") pod "811b9aae-8c34-47bd-b30b-8a6d8f957576" (UID: "811b9aae-8c34-47bd-b30b-8a6d8f957576"). InnerVolumeSpecName "kube-api-access-pfglq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:33:44.833681 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:44.833648    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfglq\" (UniqueName: \"kubernetes.io/projected/811b9aae-8c34-47bd-b30b-8a6d8f957576-kube-api-access-pfglq\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:33:44.833681 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:44.833677    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/811b9aae-8c34-47bd-b30b-8a6d8f957576-plugins-config-volume\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:33:45.452271 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:45.452227    2580 generic.go:356] "Generic (PLEG): container finished" podID="811b9aae-8c34-47bd-b30b-8a6d8f957576" containerID="081cb28fdb93ca6eaaab3e9d8f8052a9bd2fe0acf46f10bc376642a3292d1e8a" exitCode=0
Sep 11 20:33:45.452765 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:45.452282    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g"
Sep 11 20:33:45.452765 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:45.452310    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g" event={"ID":"811b9aae-8c34-47bd-b30b-8a6d8f957576","Type":"ContainerDied","Data":"081cb28fdb93ca6eaaab3e9d8f8052a9bd2fe0acf46f10bc376642a3292d1e8a"}
Sep 11 20:33:45.452765 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:45.452383    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g" event={"ID":"811b9aae-8c34-47bd-b30b-8a6d8f957576","Type":"ContainerDied","Data":"cafee661f41919ebe384bc88d594e89824043dd57dd918fa763e9fa0d28a36d1"}
Sep 11 20:33:45.452765 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:45.452400    2580 scope.go:117] "RemoveContainer" containerID="081cb28fdb93ca6eaaab3e9d8f8052a9bd2fe0acf46f10bc376642a3292d1e8a"
Sep 11 20:33:45.461939 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:45.461923    2580 scope.go:117] "RemoveContainer" containerID="081cb28fdb93ca6eaaab3e9d8f8052a9bd2fe0acf46f10bc376642a3292d1e8a"
Sep 11 20:33:45.462175 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:33:45.462159    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"081cb28fdb93ca6eaaab3e9d8f8052a9bd2fe0acf46f10bc376642a3292d1e8a\": container with ID starting with 081cb28fdb93ca6eaaab3e9d8f8052a9bd2fe0acf46f10bc376642a3292d1e8a not found: ID does not exist" containerID="081cb28fdb93ca6eaaab3e9d8f8052a9bd2fe0acf46f10bc376642a3292d1e8a"
Sep 11 20:33:45.462220 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:45.462182    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"081cb28fdb93ca6eaaab3e9d8f8052a9bd2fe0acf46f10bc376642a3292d1e8a"} err="failed to get container status \"081cb28fdb93ca6eaaab3e9d8f8052a9bd2fe0acf46f10bc376642a3292d1e8a\": rpc error: code = NotFound desc = could not find container \"081cb28fdb93ca6eaaab3e9d8f8052a9bd2fe0acf46f10bc376642a3292d1e8a\": container with ID starting with 081cb28fdb93ca6eaaab3e9d8f8052a9bd2fe0acf46f10bc376642a3292d1e8a not found: ID does not exist"
Sep 11 20:33:45.473698 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:45.473676    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g"]
Sep 11 20:33:45.477065 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:45.477042    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6453d-6987874b79-bjj6g"]
Sep 11 20:33:46.441688 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:46.441653    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c4ddfd2-7320-40ae-ab61-9c7d86489c63" path="/var/lib/kubelet/pods/6c4ddfd2-7320-40ae-ab61-9c7d86489c63/volumes"
Sep 11 20:33:46.442030 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:46.442017    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="811b9aae-8c34-47bd-b30b-8a6d8f957576" path="/var/lib/kubelet/pods/811b9aae-8c34-47bd-b30b-8a6d8f957576/volumes"
Sep 11 20:33:47.601280 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:47.601239    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:33:51.252618 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:51.252584    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fcb0e4-cjkcb"]
Sep 11 20:33:51.253128 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:51.253107    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="811b9aae-8c34-47bd-b30b-8a6d8f957576" containerName="payload-processing"
Sep 11 20:33:51.253208 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:51.253132    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="811b9aae-8c34-47bd-b30b-8a6d8f957576" containerName="payload-processing"
Sep 11 20:33:51.253208 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:51.253156    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c4ddfd2-7320-40ae-ab61-9c7d86489c63" containerName="revoke-keys"
Sep 11 20:33:51.253208 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:51.253165    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c4ddfd2-7320-40ae-ab61-9c7d86489c63" containerName="revoke-keys"
Sep 11 20:33:51.253362 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:51.253266    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="811b9aae-8c34-47bd-b30b-8a6d8f957576" containerName="payload-processing"
Sep 11 20:33:51.253362 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:51.253280    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c4ddfd2-7320-40ae-ab61-9c7d86489c63" containerName="revoke-keys"
Sep 11 20:33:51.257371 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:51.257349    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fcb0e4-cjkcb"
Sep 11 20:33:51.259715 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:51.259683    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:33:51.259990 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:51.259728    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7dbs5\""
Sep 11 20:33:51.264169 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:51.264146    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fcb0e4-cjkcb"]
Sep 11 20:33:51.288483 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:51.288451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a6188f-4a74-4c0f-ae86-4470df2c1cfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fcb0e4-cjkcb\" (UID: \"70a6188f-4a74-4c0f-ae86-4470df2c1cfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fcb0e4-cjkcb"
Sep 11 20:33:51.389027 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:51.388989    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a6188f-4a74-4c0f-ae86-4470df2c1cfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fcb0e4-cjkcb\" (UID: \"70a6188f-4a74-4c0f-ae86-4470df2c1cfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fcb0e4-cjkcb"
Sep 11 20:33:51.389554 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:51.389527    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a6188f-4a74-4c0f-ae86-4470df2c1cfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fcb0e4-cjkcb\" (UID: \"70a6188f-4a74-4c0f-ae86-4470df2c1cfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fcb0e4-cjkcb"
Sep 11 20:33:51.569553 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:51.569433    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fcb0e4-cjkcb"
Sep 11 20:33:51.838030 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:51.838004    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fcb0e4-cjkcb"]
Sep 11 20:33:51.839857 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:33:51.839825    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a6188f_4a74_4c0f_ae86_4470df2c1cfa.slice/crio-5831e8ff589e6b75364361296b63e4ef450a6925bdd862a55d787e7aebbdf99f WatchSource:0}: Error finding container 5831e8ff589e6b75364361296b63e4ef450a6925bdd862a55d787e7aebbdf99f: Status 404 returned error can't find the container with id 5831e8ff589e6b75364361296b63e4ef450a6925bdd862a55d787e7aebbdf99f
Sep 11 20:33:52.490666 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:52.490627    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fcb0e4-cjkcb" event={"ID":"70a6188f-4a74-4c0f-ae86-4470df2c1cfa","Type":"ContainerStarted","Data":"73974d98b293d6ae6dcde661a3f0ec290e20500756a7a2b25bfda5bcd6976009"}
Sep 11 20:33:52.491027 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:52.490689    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fcb0e4-cjkcb" event={"ID":"70a6188f-4a74-4c0f-ae86-4470df2c1cfa","Type":"ContainerStarted","Data":"5831e8ff589e6b75364361296b63e4ef450a6925bdd862a55d787e7aebbdf99f"}
Sep 11 20:33:52.505653 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:52.505604    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fcb0e4-cjkcb" podStartSLOduration=1.5055827769999999 podStartE2EDuration="1.505582777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:33:52.504446202 +0000 UTC m=+2002.496608488" watchObservedRunningTime="2026-09-11 20:33:52.505582777 +0000 UTC m=+2002.497745047"
Sep 11 20:33:53.497601 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:53.497566    2580 generic.go:356] "Generic (PLEG): container finished" podID="70a6188f-4a74-4c0f-ae86-4470df2c1cfa" containerID="73974d98b293d6ae6dcde661a3f0ec290e20500756a7a2b25bfda5bcd6976009" exitCode=0
Sep 11 20:33:53.498026 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:53.497651    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fcb0e4-cjkcb" event={"ID":"70a6188f-4a74-4c0f-ae86-4470df2c1cfa","Type":"ContainerDied","Data":"73974d98b293d6ae6dcde661a3f0ec290e20500756a7a2b25bfda5bcd6976009"}
Sep 11 20:33:54.630862 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:54.630838    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fcb0e4-cjkcb"
Sep 11 20:33:54.717085 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:54.717050    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a6188f-4a74-4c0f-ae86-4470df2c1cfa-maas-api-service-ca\") pod \"70a6188f-4a74-4c0f-ae86-4470df2c1cfa\" (UID: \"70a6188f-4a74-4c0f-ae86-4470df2c1cfa\") "
Sep 11 20:33:54.717417 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:54.717395    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a6188f-4a74-4c0f-ae86-4470df2c1cfa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "70a6188f-4a74-4c0f-ae86-4470df2c1cfa" (UID: "70a6188f-4a74-4c0f-ae86-4470df2c1cfa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:33:54.818116 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:54.818038    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a6188f-4a74-4c0f-ae86-4470df2c1cfa-maas-api-service-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:33:55.509870 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:55.509838    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fcb0e4-cjkcb"
Sep 11 20:33:55.510052 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:55.509832    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fcb0e4-cjkcb" event={"ID":"70a6188f-4a74-4c0f-ae86-4470df2c1cfa","Type":"ContainerDied","Data":"5831e8ff589e6b75364361296b63e4ef450a6925bdd862a55d787e7aebbdf99f"}
Sep 11 20:33:55.510052 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:33:55.509972    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5831e8ff589e6b75364361296b63e4ef450a6925bdd862a55d787e7aebbdf99f"
Sep 11 20:34:01.710349 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:01.710316    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fcb0e4-29819310-r95pm"]
Sep 11 20:34:01.713873 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:01.713843    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fcb0e4-29819310-r95pm"]
Sep 11 20:34:01.895970 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:01.895937    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fcb0e4-cjkcb"]
Sep 11 20:34:01.898158 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:01.898135    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fcb0e4-cjkcb"]
Sep 11 20:34:02.441310 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:02.441268    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70a6188f-4a74-4c0f-ae86-4470df2c1cfa" path="/var/lib/kubelet/pods/70a6188f-4a74-4c0f-ae86-4470df2c1cfa/volumes"
Sep 11 20:34:02.441648 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:02.441629    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcebaf46-aab1-4b87-b80d-d1517576b19a" path="/var/lib/kubelet/pods/dcebaf46-aab1-4b87-b80d-d1517576b19a/volumes"
Sep 11 20:34:04.798730 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:04.798698    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:34:08.543541 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:08.543511    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-092e1a-pdt4z"]
Sep 11 20:34:08.543984 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:08.543844    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70a6188f-4a74-4c0f-ae86-4470df2c1cfa" containerName="revoke-keys"
Sep 11 20:34:08.543984 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:08.543856    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="70a6188f-4a74-4c0f-ae86-4470df2c1cfa" containerName="revoke-keys"
Sep 11 20:34:08.543984 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:08.543913    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="70a6188f-4a74-4c0f-ae86-4470df2c1cfa" containerName="revoke-keys"
Sep 11 20:34:08.546753 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:08.546735    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-092e1a-pdt4z"
Sep 11 20:34:08.549902 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:08.549870    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 20:34:08.549902 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:08.549896    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7dbs5\""
Sep 11 20:34:08.554730 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:08.554709    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-092e1a-pdt4z"]
Sep 11 20:34:08.640425 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:08.640390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff1b9250-a142-4800-ac52-3060ea2302d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-092e1a-pdt4z\" (UID: \"ff1b9250-a142-4800-ac52-3060ea2302d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-092e1a-pdt4z"
Sep 11 20:34:08.741313 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:08.741272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff1b9250-a142-4800-ac52-3060ea2302d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-092e1a-pdt4z\" (UID: \"ff1b9250-a142-4800-ac52-3060ea2302d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-092e1a-pdt4z"
Sep 11 20:34:08.741899 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:08.741878    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff1b9250-a142-4800-ac52-3060ea2302d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-092e1a-pdt4z\" (UID: \"ff1b9250-a142-4800-ac52-3060ea2302d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-092e1a-pdt4z"
Sep 11 20:34:08.858480 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:08.858399    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-092e1a-pdt4z"
Sep 11 20:34:09.132141 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:09.132044    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-092e1a-pdt4z"]
Sep 11 20:34:09.134916 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:34:09.134891    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff1b9250_a142_4800_ac52_3060ea2302d9.slice/crio-4b7cb3b494ff9c74dc84def49fdc6526c4fd412e28b619bd63219275d5415b16 WatchSource:0}: Error finding container 4b7cb3b494ff9c74dc84def49fdc6526c4fd412e28b619bd63219275d5415b16: Status 404 returned error can't find the container with id 4b7cb3b494ff9c74dc84def49fdc6526c4fd412e28b619bd63219275d5415b16
Sep 11 20:34:09.589991 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:09.589947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-092e1a-pdt4z" event={"ID":"ff1b9250-a142-4800-ac52-3060ea2302d9","Type":"ContainerStarted","Data":"4fb21cacf16f3c0f3d3dc6f9ca979e57fcfd0e6bcf622158058e964a3f3974bc"}
Sep 11 20:34:09.590350 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:09.590008    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-092e1a-pdt4z" event={"ID":"ff1b9250-a142-4800-ac52-3060ea2302d9","Type":"ContainerStarted","Data":"4b7cb3b494ff9c74dc84def49fdc6526c4fd412e28b619bd63219275d5415b16"}
Sep 11 20:34:09.607212 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:09.607168    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-092e1a-pdt4z" podStartSLOduration=1.607149681 podStartE2EDuration="1.607149681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:34:09.604415893 +0000 UTC m=+2019.596578163" watchObservedRunningTime="2026-09-11 20:34:09.607149681 +0000 UTC m=+2019.599311951"
Sep 11 20:34:10.597304 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:10.597270    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff1b9250-a142-4800-ac52-3060ea2302d9" containerID="4fb21cacf16f3c0f3d3dc6f9ca979e57fcfd0e6bcf622158058e964a3f3974bc" exitCode=0
Sep 11 20:34:10.597702 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:10.597325    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-092e1a-pdt4z" event={"ID":"ff1b9250-a142-4800-ac52-3060ea2302d9","Type":"ContainerDied","Data":"4fb21cacf16f3c0f3d3dc6f9ca979e57fcfd0e6bcf622158058e964a3f3974bc"}
Sep 11 20:34:11.730409 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:11.730384    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-092e1a-pdt4z"
Sep 11 20:34:11.866852 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:11.866755    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff1b9250-a142-4800-ac52-3060ea2302d9-maas-api-service-ca\") pod \"ff1b9250-a142-4800-ac52-3060ea2302d9\" (UID: \"ff1b9250-a142-4800-ac52-3060ea2302d9\") "
Sep 11 20:34:11.867163 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:11.867137    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff1b9250-a142-4800-ac52-3060ea2302d9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ff1b9250-a142-4800-ac52-3060ea2302d9" (UID: "ff1b9250-a142-4800-ac52-3060ea2302d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:34:11.967779 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:11.967743    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff1b9250-a142-4800-ac52-3060ea2302d9-maas-api-service-ca\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:34:12.609234 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:12.609205    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-092e1a-pdt4z"
Sep 11 20:34:12.609416 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:12.609246    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-092e1a-pdt4z" event={"ID":"ff1b9250-a142-4800-ac52-3060ea2302d9","Type":"ContainerDied","Data":"4b7cb3b494ff9c74dc84def49fdc6526c4fd412e28b619bd63219275d5415b16"}
Sep 11 20:34:12.609416 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:12.609291    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b7cb3b494ff9c74dc84def49fdc6526c4fd412e28b619bd63219275d5415b16"
Sep 11 20:34:18.933439 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:18.933405    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-092e1a-29819310-6cdwt"]
Sep 11 20:34:18.938176 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:18.938149    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-092e1a-29819310-6cdwt"]
Sep 11 20:34:18.960122 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:18.960091    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb"]
Sep 11 20:34:18.960355 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:18.960328    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb" podUID="cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e" containerName="payload-processing" containerID="cri-o://5209a22fb72eb3e71ff774c989a4e709bb3b760315fcbb41e9cb3be583b9a211" gracePeriod=30
Sep 11 20:34:18.967467 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:18.967444    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm"]
Sep 11 20:34:18.967669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:18.967648    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm" podUID="dcda5f1e-d036-45a2-bf00-60539701c98e" containerName="payload-pre-processing" containerID="cri-o://c5f9f737b40520cafc6ea66a89e50d758ce0f9d1a132263a5b92c4e41c18087c" gracePeriod=30
Sep 11 20:34:19.129272 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.129231    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-092e1a-pdt4z"]
Sep 11 20:34:19.132109 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.132049    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-092e1a-pdt4z"]
Sep 11 20:34:19.238543 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.238517    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb"
Sep 11 20:34:19.241258 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.241238    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm"
Sep 11 20:34:19.328729 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.328698    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gv9f5\" (UniqueName: \"kubernetes.io/projected/dcda5f1e-d036-45a2-bf00-60539701c98e-kube-api-access-gv9f5\") pod \"dcda5f1e-d036-45a2-bf00-60539701c98e\" (UID: \"dcda5f1e-d036-45a2-bf00-60539701c98e\") "
Sep 11 20:34:19.328892 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.328769    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e-plugins-config-volume\") pod \"cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e\" (UID: \"cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e\") "
Sep 11 20:34:19.328892 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.328820    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8zxw\" (UniqueName: \"kubernetes.io/projected/cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e-kube-api-access-f8zxw\") pod \"cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e\" (UID: \"cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e\") "
Sep 11 20:34:19.328892 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.328847    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcda5f1e-d036-45a2-bf00-60539701c98e-plugins-config-volume\") pod \"dcda5f1e-d036-45a2-bf00-60539701c98e\" (UID: \"dcda5f1e-d036-45a2-bf00-60539701c98e\") "
Sep 11 20:34:19.329168 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.329145    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e" (UID: "cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:34:19.329239 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.329193    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcda5f1e-d036-45a2-bf00-60539701c98e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dcda5f1e-d036-45a2-bf00-60539701c98e" (UID: "dcda5f1e-d036-45a2-bf00-60539701c98e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:34:19.330902 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.330876    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e-kube-api-access-f8zxw" (OuterVolumeSpecName: "kube-api-access-f8zxw") pod "cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e" (UID: "cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e"). InnerVolumeSpecName "kube-api-access-f8zxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:34:19.330989 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.330910    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcda5f1e-d036-45a2-bf00-60539701c98e-kube-api-access-gv9f5" (OuterVolumeSpecName: "kube-api-access-gv9f5") pod "dcda5f1e-d036-45a2-bf00-60539701c98e" (UID: "dcda5f1e-d036-45a2-bf00-60539701c98e"). InnerVolumeSpecName "kube-api-access-gv9f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:34:19.430256 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.430220    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e-plugins-config-volume\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:34:19.430256 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.430249    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8zxw\" (UniqueName: \"kubernetes.io/projected/cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e-kube-api-access-f8zxw\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:34:19.430256 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.430260    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcda5f1e-d036-45a2-bf00-60539701c98e-plugins-config-volume\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:34:19.430455 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.430268    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv9f5\" (UniqueName: \"kubernetes.io/projected/dcda5f1e-d036-45a2-bf00-60539701c98e-kube-api-access-gv9f5\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:34:19.649308 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.649227    2580 generic.go:356] "Generic (PLEG): container finished" podID="dcda5f1e-d036-45a2-bf00-60539701c98e" containerID="c5f9f737b40520cafc6ea66a89e50d758ce0f9d1a132263a5b92c4e41c18087c" exitCode=0
Sep 11 20:34:19.649308 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.649273    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm"
Sep 11 20:34:19.649528 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.649316    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm" event={"ID":"dcda5f1e-d036-45a2-bf00-60539701c98e","Type":"ContainerDied","Data":"c5f9f737b40520cafc6ea66a89e50d758ce0f9d1a132263a5b92c4e41c18087c"}
Sep 11 20:34:19.649528 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.649376    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm" event={"ID":"dcda5f1e-d036-45a2-bf00-60539701c98e","Type":"ContainerDied","Data":"29190ae698e856cc26cab565316850413fce6d880c5f97ed4ec6e1a016f4e9da"}
Sep 11 20:34:19.649528 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.649410    2580 scope.go:117] "RemoveContainer" containerID="c5f9f737b40520cafc6ea66a89e50d758ce0f9d1a132263a5b92c4e41c18087c"
Sep 11 20:34:19.650766 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.650738    2580 generic.go:356] "Generic (PLEG): container finished" podID="cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e" containerID="5209a22fb72eb3e71ff774c989a4e709bb3b760315fcbb41e9cb3be583b9a211" exitCode=0
Sep 11 20:34:19.650766 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.650762    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb" event={"ID":"cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e","Type":"ContainerDied","Data":"5209a22fb72eb3e71ff774c989a4e709bb3b760315fcbb41e9cb3be583b9a211"}
Sep 11 20:34:19.650947 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.650789    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb"
Sep 11 20:34:19.650947 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.650803    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb" event={"ID":"cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e","Type":"ContainerDied","Data":"91d85c7c73a7d59efe0f9c257740a2e32fd0d50a1ed13ff250f34c372471e33e"}
Sep 11 20:34:19.660738 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.660721    2580 scope.go:117] "RemoveContainer" containerID="c5f9f737b40520cafc6ea66a89e50d758ce0f9d1a132263a5b92c4e41c18087c"
Sep 11 20:34:19.661111 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:34:19.661001    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5f9f737b40520cafc6ea66a89e50d758ce0f9d1a132263a5b92c4e41c18087c\": container with ID starting with c5f9f737b40520cafc6ea66a89e50d758ce0f9d1a132263a5b92c4e41c18087c not found: ID does not exist" containerID="c5f9f737b40520cafc6ea66a89e50d758ce0f9d1a132263a5b92c4e41c18087c"
Sep 11 20:34:19.661111 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.661032    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5f9f737b40520cafc6ea66a89e50d758ce0f9d1a132263a5b92c4e41c18087c"} err="failed to get container status \"c5f9f737b40520cafc6ea66a89e50d758ce0f9d1a132263a5b92c4e41c18087c\": rpc error: code = NotFound desc = could not find container \"c5f9f737b40520cafc6ea66a89e50d758ce0f9d1a132263a5b92c4e41c18087c\": container with ID starting with c5f9f737b40520cafc6ea66a89e50d758ce0f9d1a132263a5b92c4e41c18087c not found: ID does not exist"
Sep 11 20:34:19.661111 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.661050    2580 scope.go:117] "RemoveContainer" containerID="5209a22fb72eb3e71ff774c989a4e709bb3b760315fcbb41e9cb3be583b9a211"
Sep 11 20:34:19.669547 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.669530    2580 scope.go:117] "RemoveContainer" containerID="5209a22fb72eb3e71ff774c989a4e709bb3b760315fcbb41e9cb3be583b9a211"
Sep 11 20:34:19.669779 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:34:19.669762    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5209a22fb72eb3e71ff774c989a4e709bb3b760315fcbb41e9cb3be583b9a211\": container with ID starting with 5209a22fb72eb3e71ff774c989a4e709bb3b760315fcbb41e9cb3be583b9a211 not found: ID does not exist" containerID="5209a22fb72eb3e71ff774c989a4e709bb3b760315fcbb41e9cb3be583b9a211"
Sep 11 20:34:19.669822 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.669786    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5209a22fb72eb3e71ff774c989a4e709bb3b760315fcbb41e9cb3be583b9a211"} err="failed to get container status \"5209a22fb72eb3e71ff774c989a4e709bb3b760315fcbb41e9cb3be583b9a211\": rpc error: code = NotFound desc = could not find container \"5209a22fb72eb3e71ff774c989a4e709bb3b760315fcbb41e9cb3be583b9a211\": container with ID starting with 5209a22fb72eb3e71ff774c989a4e709bb3b760315fcbb41e9cb3be583b9a211 not found: ID does not exist"
Sep 11 20:34:19.675746 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.675726    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb"]
Sep 11 20:34:19.681412 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.681391    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-092e1a-7c6446f69-2cqpb"]
Sep 11 20:34:19.691115 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.691084    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm"]
Sep 11 20:34:19.693031 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:19.693010    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-092e1a-6ccf898894-r7mlm"]
Sep 11 20:34:20.440298 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:20.440267    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b15257a-7c74-4b1d-a88e-4cbe198dd569" path="/var/lib/kubelet/pods/2b15257a-7c74-4b1d-a88e-4cbe198dd569/volumes"
Sep 11 20:34:20.440669 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:20.440591    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e" path="/var/lib/kubelet/pods/cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e/volumes"
Sep 11 20:34:20.440918 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:20.440906    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcda5f1e-d036-45a2-bf00-60539701c98e" path="/var/lib/kubelet/pods/dcda5f1e-d036-45a2-bf00-60539701c98e/volumes"
Sep 11 20:34:20.441186 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:20.441175    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff1b9250-a142-4800-ac52-3060ea2302d9" path="/var/lib/kubelet/pods/ff1b9250-a142-4800-ac52-3060ea2302d9/volumes"
Sep 11 20:34:21.605237 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:21.605194    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:34:29.500196 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:34:29.500155    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:35:51.901700 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:35:51.901662    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:36:16.804764 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:36:16.804723    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:36:24.399672 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:36:24.399638    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:36:31.075456 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:36:31.075423    2580 scope.go:117] "RemoveContainer" containerID="c169be4b91ec8dbb0452f06ec1693f626b9049b82d85e2281e17f654e08fa8d0"
Sep 11 20:36:31.084072 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:36:31.084054    2580 scope.go:117] "RemoveContainer" containerID="d089cf3f0ccf4103e5d31f3d95610cec2fe3204d570906b770544eb9757cbe4d"
Sep 11 20:36:31.304978 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:36:31.304939    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:36:38.799267 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:36:38.799225    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:36:53.406333 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:36:53.406298    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:37:01.797525 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:37:01.797477    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:38:28.998044 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:38:28.998005    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:38:31.114707 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:38:31.114678    2580 scope.go:117] "RemoveContainer" containerID="4b93f2b87d02916cb53c843c88dddca39e6f6bd6cb806c3722c56248dc21b229"
Sep 11 20:38:31.122994 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:38:31.122975    2580 scope.go:117] "RemoveContainer" containerID="f1f394ad9624746c91f4020c2d4c79471a175af53620fe1f4df01d338ed2e1e6"
Sep 11 20:38:37.509024 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:38:37.508986    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:38:45.304284 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:38:45.304239    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:38:51.804092 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:38:51.804042    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:39:01.178366 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.178331    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"]
Sep 11 20:39:01.178864 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.178844    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcda5f1e-d036-45a2-bf00-60539701c98e" containerName="payload-pre-processing"
Sep 11 20:39:01.178941 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.178868    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcda5f1e-d036-45a2-bf00-60539701c98e" containerName="payload-pre-processing"
Sep 11 20:39:01.178941 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.178885    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff1b9250-a142-4800-ac52-3060ea2302d9" containerName="revoke-keys"
Sep 11 20:39:01.178941 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.178894    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff1b9250-a142-4800-ac52-3060ea2302d9" containerName="revoke-keys"
Sep 11 20:39:01.178941 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.178910    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e" containerName="payload-processing"
Sep 11 20:39:01.178941 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.178918    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e" containerName="payload-processing"
Sep 11 20:39:01.179248 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.179010    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff1b9250-a142-4800-ac52-3060ea2302d9" containerName="revoke-keys"
Sep 11 20:39:01.179248 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.179024    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcda5f1e-d036-45a2-bf00-60539701c98e" containerName="payload-pre-processing"
Sep 11 20:39:01.179248 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.179039    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfc34adf-5bdf-44d9-a36d-7eb4c4896c3e" containerName="payload-processing"
Sep 11 20:39:01.182268 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.182244    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:01.184575 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.184553    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-32def012-kserve-self-signed-certs\""
Sep 11 20:39:01.192687 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.192665    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"]
Sep 11 20:39:01.286653 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.286616    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d618fef0-255b-431f-8aee-8d465c4c5473-tmp-dir\") pod \"e2e-same-id-b-32def012-kserve-6456485d75-bxjhh\" (UID: \"d618fef0-255b-431f-8aee-8d465c4c5473\") " pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:01.286820 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.286679    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d618fef0-255b-431f-8aee-8d465c4c5473-home\") pod \"e2e-same-id-b-32def012-kserve-6456485d75-bxjhh\" (UID: \"d618fef0-255b-431f-8aee-8d465c4c5473\") " pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:01.286820 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.286728    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d618fef0-255b-431f-8aee-8d465c4c5473-dshm\") pod \"e2e-same-id-b-32def012-kserve-6456485d75-bxjhh\" (UID: \"d618fef0-255b-431f-8aee-8d465c4c5473\") " pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:01.286820 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.286760    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v42rr\" (UniqueName: \"kubernetes.io/projected/d618fef0-255b-431f-8aee-8d465c4c5473-kube-api-access-v42rr\") pod \"e2e-same-id-b-32def012-kserve-6456485d75-bxjhh\" (UID: \"d618fef0-255b-431f-8aee-8d465c4c5473\") " pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:01.286820 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.286782    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d618fef0-255b-431f-8aee-8d465c4c5473-model-cache\") pod \"e2e-same-id-b-32def012-kserve-6456485d75-bxjhh\" (UID: \"d618fef0-255b-431f-8aee-8d465c4c5473\") " pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:01.286820 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.286819    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d618fef0-255b-431f-8aee-8d465c4c5473-tls-certs\") pod \"e2e-same-id-b-32def012-kserve-6456485d75-bxjhh\" (UID: \"d618fef0-255b-431f-8aee-8d465c4c5473\") " pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:01.388062 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.388018    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d618fef0-255b-431f-8aee-8d465c4c5473-tls-certs\") pod \"e2e-same-id-b-32def012-kserve-6456485d75-bxjhh\" (UID: \"d618fef0-255b-431f-8aee-8d465c4c5473\") " pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:01.388313 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.388072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d618fef0-255b-431f-8aee-8d465c4c5473-tmp-dir\") pod \"e2e-same-id-b-32def012-kserve-6456485d75-bxjhh\" (UID: \"d618fef0-255b-431f-8aee-8d465c4c5473\") " pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:01.388313 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.388116    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d618fef0-255b-431f-8aee-8d465c4c5473-home\") pod \"e2e-same-id-b-32def012-kserve-6456485d75-bxjhh\" (UID: \"d618fef0-255b-431f-8aee-8d465c4c5473\") " pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:01.388313 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.388137    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d618fef0-255b-431f-8aee-8d465c4c5473-dshm\") pod \"e2e-same-id-b-32def012-kserve-6456485d75-bxjhh\" (UID: \"d618fef0-255b-431f-8aee-8d465c4c5473\") " pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:01.388313 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.388155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v42rr\" (UniqueName: \"kubernetes.io/projected/d618fef0-255b-431f-8aee-8d465c4c5473-kube-api-access-v42rr\") pod \"e2e-same-id-b-32def012-kserve-6456485d75-bxjhh\" (UID: \"d618fef0-255b-431f-8aee-8d465c4c5473\") " pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:01.388313 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.388175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d618fef0-255b-431f-8aee-8d465c4c5473-model-cache\") pod \"e2e-same-id-b-32def012-kserve-6456485d75-bxjhh\" (UID: \"d618fef0-255b-431f-8aee-8d465c4c5473\") " pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:01.388638 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.388592    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d618fef0-255b-431f-8aee-8d465c4c5473-home\") pod \"e2e-same-id-b-32def012-kserve-6456485d75-bxjhh\" (UID: \"d618fef0-255b-431f-8aee-8d465c4c5473\") " pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:01.388638 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.388627    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d618fef0-255b-431f-8aee-8d465c4c5473-tmp-dir\") pod \"e2e-same-id-b-32def012-kserve-6456485d75-bxjhh\" (UID: \"d618fef0-255b-431f-8aee-8d465c4c5473\") " pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:01.388736 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.388658    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d618fef0-255b-431f-8aee-8d465c4c5473-model-cache\") pod \"e2e-same-id-b-32def012-kserve-6456485d75-bxjhh\" (UID: \"d618fef0-255b-431f-8aee-8d465c4c5473\") " pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:01.390449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.390418    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d618fef0-255b-431f-8aee-8d465c4c5473-dshm\") pod \"e2e-same-id-b-32def012-kserve-6456485d75-bxjhh\" (UID: \"d618fef0-255b-431f-8aee-8d465c4c5473\") " pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:01.390673 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.390657    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d618fef0-255b-431f-8aee-8d465c4c5473-tls-certs\") pod \"e2e-same-id-b-32def012-kserve-6456485d75-bxjhh\" (UID: \"d618fef0-255b-431f-8aee-8d465c4c5473\") " pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:01.395466 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.395443    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v42rr\" (UniqueName: \"kubernetes.io/projected/d618fef0-255b-431f-8aee-8d465c4c5473-kube-api-access-v42rr\") pod \"e2e-same-id-b-32def012-kserve-6456485d75-bxjhh\" (UID: \"d618fef0-255b-431f-8aee-8d465c4c5473\") " pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:01.493569 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.493527    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:01.769348 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:01.769267    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"]
Sep 11 20:39:01.772181 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:39:01.772150    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd618fef0_255b_431f_8aee_8d465c4c5473.slice/crio-f068b2c5ff56f7e9450e2c1ade41895a71e8d37099993fb6188ebda010807bd6 WatchSource:0}: Error finding container f068b2c5ff56f7e9450e2c1ade41895a71e8d37099993fb6188ebda010807bd6: Status 404 returned error can't find the container with id f068b2c5ff56f7e9450e2c1ade41895a71e8d37099993fb6188ebda010807bd6
Sep 11 20:39:02.121171 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:02.121078    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh" event={"ID":"d618fef0-255b-431f-8aee-8d465c4c5473","Type":"ContainerStarted","Data":"a774734e0399ccb7c8125e04a70d19149987da90687015bbc382c9b6d8d5a263"}
Sep 11 20:39:02.121171 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:02.121122    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh" event={"ID":"d618fef0-255b-431f-8aee-8d465c4c5473","Type":"ContainerStarted","Data":"f068b2c5ff56f7e9450e2c1ade41895a71e8d37099993fb6188ebda010807bd6"}
Sep 11 20:39:02.139781 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:02.139727    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh" podStartSLOduration=1.139712998 podStartE2EDuration="1.139712998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:39:02.137409224 +0000 UTC m=+2312.129571493" watchObservedRunningTime="2026-09-11 20:39:02.139712998 +0000 UTC m=+2312.131875268"
Sep 11 20:39:04.113469 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:04.113431    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:39:11.494345 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:11.494310    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:11.494801 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:11.494514    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:11.506947 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:11.506922    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:12.184082 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:12.184051    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:28.662338 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:28.662302    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"]
Sep 11 20:39:29.306655 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:29.306621    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:39:38.703372 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:38.703332    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:39:43.681788 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:43.681728    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh" podUID="d618fef0-255b-431f-8aee-8d465c4c5473" containerName="main" containerID="cri-o://a774734e0399ccb7c8125e04a70d19149987da90687015bbc382c9b6d8d5a263" gracePeriod=45
Sep 11 20:39:43.926903 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:43.926877    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:44.058623 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.058590    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d618fef0-255b-431f-8aee-8d465c4c5473-tmp-dir\") pod \"d618fef0-255b-431f-8aee-8d465c4c5473\" (UID: \"d618fef0-255b-431f-8aee-8d465c4c5473\") "
Sep 11 20:39:44.058822 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.058644    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v42rr\" (UniqueName: \"kubernetes.io/projected/d618fef0-255b-431f-8aee-8d465c4c5473-kube-api-access-v42rr\") pod \"d618fef0-255b-431f-8aee-8d465c4c5473\" (UID: \"d618fef0-255b-431f-8aee-8d465c4c5473\") "
Sep 11 20:39:44.058822 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.058683    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d618fef0-255b-431f-8aee-8d465c4c5473-dshm\") pod \"d618fef0-255b-431f-8aee-8d465c4c5473\" (UID: \"d618fef0-255b-431f-8aee-8d465c4c5473\") "
Sep 11 20:39:44.058822 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.058724    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d618fef0-255b-431f-8aee-8d465c4c5473-model-cache\") pod \"d618fef0-255b-431f-8aee-8d465c4c5473\" (UID: \"d618fef0-255b-431f-8aee-8d465c4c5473\") "
Sep 11 20:39:44.058822 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.058769    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d618fef0-255b-431f-8aee-8d465c4c5473-home\") pod \"d618fef0-255b-431f-8aee-8d465c4c5473\" (UID: \"d618fef0-255b-431f-8aee-8d465c4c5473\") "
Sep 11 20:39:44.059036 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.058830    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d618fef0-255b-431f-8aee-8d465c4c5473-tls-certs\") pod \"d618fef0-255b-431f-8aee-8d465c4c5473\" (UID: \"d618fef0-255b-431f-8aee-8d465c4c5473\") "
Sep 11 20:39:44.059036 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.058952    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d618fef0-255b-431f-8aee-8d465c4c5473-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d618fef0-255b-431f-8aee-8d465c4c5473" (UID: "d618fef0-255b-431f-8aee-8d465c4c5473"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:39:44.059036 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.059008    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d618fef0-255b-431f-8aee-8d465c4c5473-model-cache" (OuterVolumeSpecName: "model-cache") pod "d618fef0-255b-431f-8aee-8d465c4c5473" (UID: "d618fef0-255b-431f-8aee-8d465c4c5473"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:39:44.059191 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.059058    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d618fef0-255b-431f-8aee-8d465c4c5473-home" (OuterVolumeSpecName: "home") pod "d618fef0-255b-431f-8aee-8d465c4c5473" (UID: "d618fef0-255b-431f-8aee-8d465c4c5473"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:39:44.059191 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.059159    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d618fef0-255b-431f-8aee-8d465c4c5473-tmp-dir\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:39:44.059191 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.059179    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d618fef0-255b-431f-8aee-8d465c4c5473-model-cache\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:39:44.059309 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.059192    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d618fef0-255b-431f-8aee-8d465c4c5473-home\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:39:44.061045 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.061012    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d618fef0-255b-431f-8aee-8d465c4c5473-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d618fef0-255b-431f-8aee-8d465c4c5473" (UID: "d618fef0-255b-431f-8aee-8d465c4c5473"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:39:44.061045 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.061030    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d618fef0-255b-431f-8aee-8d465c4c5473-kube-api-access-v42rr" (OuterVolumeSpecName: "kube-api-access-v42rr") pod "d618fef0-255b-431f-8aee-8d465c4c5473" (UID: "d618fef0-255b-431f-8aee-8d465c4c5473"). InnerVolumeSpecName "kube-api-access-v42rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:39:44.061212 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.061077    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d618fef0-255b-431f-8aee-8d465c4c5473-dshm" (OuterVolumeSpecName: "dshm") pod "d618fef0-255b-431f-8aee-8d465c4c5473" (UID: "d618fef0-255b-431f-8aee-8d465c4c5473"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:39:44.159801 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.159766    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v42rr\" (UniqueName: \"kubernetes.io/projected/d618fef0-255b-431f-8aee-8d465c4c5473-kube-api-access-v42rr\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:39:44.159801 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.159795    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d618fef0-255b-431f-8aee-8d465c4c5473-dshm\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:39:44.159801 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.159806    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d618fef0-255b-431f-8aee-8d465c4c5473-tls-certs\") on node \"ip-10-0-128-209.ec2.internal\" DevicePath \"\""
Sep 11 20:39:44.342477 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.342386    2580 generic.go:356] "Generic (PLEG): container finished" podID="d618fef0-255b-431f-8aee-8d465c4c5473" containerID="a774734e0399ccb7c8125e04a70d19149987da90687015bbc382c9b6d8d5a263" exitCode=0
Sep 11 20:39:44.342477 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.342431    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"
Sep 11 20:39:44.342715 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.342473    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh" event={"ID":"d618fef0-255b-431f-8aee-8d465c4c5473","Type":"ContainerDied","Data":"a774734e0399ccb7c8125e04a70d19149987da90687015bbc382c9b6d8d5a263"}
Sep 11 20:39:44.342715 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.342525    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh" event={"ID":"d618fef0-255b-431f-8aee-8d465c4c5473","Type":"ContainerDied","Data":"f068b2c5ff56f7e9450e2c1ade41895a71e8d37099993fb6188ebda010807bd6"}
Sep 11 20:39:44.342715 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.342544    2580 scope.go:117] "RemoveContainer" containerID="a774734e0399ccb7c8125e04a70d19149987da90687015bbc382c9b6d8d5a263"
Sep 11 20:39:44.351617 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.351598    2580 scope.go:117] "RemoveContainer" containerID="a774734e0399ccb7c8125e04a70d19149987da90687015bbc382c9b6d8d5a263"
Sep 11 20:39:44.351899 ip-10-0-128-209 kubenswrapper[2580]: E0911 20:39:44.351881    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a774734e0399ccb7c8125e04a70d19149987da90687015bbc382c9b6d8d5a263\": container with ID starting with a774734e0399ccb7c8125e04a70d19149987da90687015bbc382c9b6d8d5a263 not found: ID does not exist" containerID="a774734e0399ccb7c8125e04a70d19149987da90687015bbc382c9b6d8d5a263"
Sep 11 20:39:44.351961 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.351912    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a774734e0399ccb7c8125e04a70d19149987da90687015bbc382c9b6d8d5a263"} err="failed to get container status \"a774734e0399ccb7c8125e04a70d19149987da90687015bbc382c9b6d8d5a263\": rpc error: code = NotFound desc = could not find container \"a774734e0399ccb7c8125e04a70d19149987da90687015bbc382c9b6d8d5a263\": container with ID starting with a774734e0399ccb7c8125e04a70d19149987da90687015bbc382c9b6d8d5a263 not found: ID does not exist"
Sep 11 20:39:44.363396 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.363372    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"]
Sep 11 20:39:44.366418 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.366399    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-32def012-kserve-6456485d75-bxjhh"]
Sep 11 20:39:44.439951 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:44.439927    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d618fef0-255b-431f-8aee-8d465c4c5473" path="/var/lib/kubelet/pods/d618fef0-255b-431f-8aee-8d465c4c5473/volumes"
Sep 11 20:39:46.402231 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:46.402198    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:39:54.503780 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:39:54.503746    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:40:01.302243 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:01.302207    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:40:06.998903 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:06.998866    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:40:13.507028 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:13.506992    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-7kjmw"]
Sep 11 20:40:16.443606 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:16.443574    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b84cff79-88hf6_a1f9e71e-d948-4142-a345-5e8231aabf2d/authorino/0.log"
Sep 11 20:40:22.239930 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:22.239901    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fdf48ff49-24xlt_4bb46042-91b3-4506-857f-4db20fa98890/copy-manifests/0.log"
Sep 11 20:40:22.245237 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:22.245209    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fdf48ff49-24xlt_4bb46042-91b3-4506-857f-4db20fa98890/manager/0.log"
Sep 11 20:40:23.155483 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:23.155454    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819310-rtwfk_411174cb-1485-4333-ab49-ad3dc2ec2196/cleanup/0.log"
Sep 11 20:40:24.014165 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:24.014136    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b84cff79-88hf6_a1f9e71e-d948-4142-a345-5e8231aabf2d/authorino/0.log"
Sep 11 20:40:24.241760 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:24.241722    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-tx4dp_2f73cdfa-3681-4c39-915b-366711653f6f/manager/0.log"
Sep 11 20:40:24.350333 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:24.350250    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-5dnxt_e6bd712e-5be3-4bee-8974-76dff2193cac/kuadrant-console-plugin/0.log"
Sep 11 20:40:24.595625 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:24.595597    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-7kjmw_6f53db6b-18c9-46a8-801a-da9c44c8bc17/limitador/0.log"
Sep 11 20:40:25.277991 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:25.277962    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-567ff9dcd5-hrxd6_956f104b-830b-4d30-8185-d446debf54c3/kube-auth-proxy/0.log"
Sep 11 20:40:25.738803 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:25.738776    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jtmt6_76f2ec85-7849-4143-9b55-65fe0133f662/payload-processing/0.log"
Sep 11 20:40:26.723819 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:26.723792    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-7wqzj_d41da7e7-8387-4e8e-b5cc-cc45be55614a/main/0.log"
Sep 11 20:40:26.831449 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:26.831410    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-jhgm9_d5d71cd2-848a-49f5-a492-16b652abb1f3/main/0.log"
Sep 11 20:40:31.159647 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:31.159617    2580 scope.go:117] "RemoveContainer" containerID="73974d98b293d6ae6dcde661a3f0ec290e20500756a7a2b25bfda5bcd6976009"
Sep 11 20:40:31.169812 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:31.169795    2580 scope.go:117] "RemoveContainer" containerID="4fb21cacf16f3c0f3d3dc6f9ca979e57fcfd0e6bcf622158058e964a3f3974bc"
Sep 11 20:40:31.177633 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:31.177612    2580 scope.go:117] "RemoveContainer" containerID="2df2acc8f65b40631169671e64179e3bc89e4d4dea689b262c64b6f1081aada6"
Sep 11 20:40:31.695940 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:31.695905    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mjrrg/must-gather-9scff"]
Sep 11 20:40:31.696261 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:31.696225    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d618fef0-255b-431f-8aee-8d465c4c5473" containerName="main"
Sep 11 20:40:31.696261 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:31.696247    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d618fef0-255b-431f-8aee-8d465c4c5473" containerName="main"
Sep 11 20:40:31.696876 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:31.696357    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d618fef0-255b-431f-8aee-8d465c4c5473" containerName="main"
Sep 11 20:40:31.701777 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:31.701757    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mjrrg/must-gather-9scff"
Sep 11 20:40:31.704100 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:31.704078    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mjrrg\"/\"kube-root-ca.crt\""
Sep 11 20:40:31.704206 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:31.704153    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mjrrg\"/\"openshift-service-ca.crt\""
Sep 11 20:40:31.705642 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:31.705617    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mjrrg\"/\"default-dockercfg-t9mmc\""
Sep 11 20:40:31.707612 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:31.707593    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mjrrg/must-gather-9scff"]
Sep 11 20:40:31.745610 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:31.745576    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3cddeb5e-ea42-4f71-b02a-64540f746a8e-must-gather-output\") pod \"must-gather-9scff\" (UID: \"3cddeb5e-ea42-4f71-b02a-64540f746a8e\") " pod="openshift-must-gather-mjrrg/must-gather-9scff"
Sep 11 20:40:31.745762 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:31.745660    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpjvz\" (UniqueName: \"kubernetes.io/projected/3cddeb5e-ea42-4f71-b02a-64540f746a8e-kube-api-access-vpjvz\") pod \"must-gather-9scff\" (UID: \"3cddeb5e-ea42-4f71-b02a-64540f746a8e\") " pod="openshift-must-gather-mjrrg/must-gather-9scff"
Sep 11 20:40:31.846242 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:31.846212    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpjvz\" (UniqueName: \"kubernetes.io/projected/3cddeb5e-ea42-4f71-b02a-64540f746a8e-kube-api-access-vpjvz\") pod \"must-gather-9scff\" (UID: \"3cddeb5e-ea42-4f71-b02a-64540f746a8e\") " pod="openshift-must-gather-mjrrg/must-gather-9scff"
Sep 11 20:40:31.846398 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:31.846264    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3cddeb5e-ea42-4f71-b02a-64540f746a8e-must-gather-output\") pod \"must-gather-9scff\" (UID: \"3cddeb5e-ea42-4f71-b02a-64540f746a8e\") " pod="openshift-must-gather-mjrrg/must-gather-9scff"
Sep 11 20:40:31.846604 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:31.846585    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3cddeb5e-ea42-4f71-b02a-64540f746a8e-must-gather-output\") pod \"must-gather-9scff\" (UID: \"3cddeb5e-ea42-4f71-b02a-64540f746a8e\") " pod="openshift-must-gather-mjrrg/must-gather-9scff"
Sep 11 20:40:31.854203 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:31.854182    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpjvz\" (UniqueName: \"kubernetes.io/projected/3cddeb5e-ea42-4f71-b02a-64540f746a8e-kube-api-access-vpjvz\") pod \"must-gather-9scff\" (UID: \"3cddeb5e-ea42-4f71-b02a-64540f746a8e\") " pod="openshift-must-gather-mjrrg/must-gather-9scff"
Sep 11 20:40:32.012018 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:32.011942    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mjrrg/must-gather-9scff"
Sep 11 20:40:32.280109 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:32.280082    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mjrrg/must-gather-9scff"]
Sep 11 20:40:32.282421 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:40:32.282393    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3cddeb5e_ea42_4f71_b02a_64540f746a8e.slice/crio-927d698b90f7fcb142a26fe1d6af5075fcab9d68f65f3b7a74d80bf7c1cead2d WatchSource:0}: Error finding container 927d698b90f7fcb142a26fe1d6af5075fcab9d68f65f3b7a74d80bf7c1cead2d: Status 404 returned error can't find the container with id 927d698b90f7fcb142a26fe1d6af5075fcab9d68f65f3b7a74d80bf7c1cead2d
Sep 11 20:40:32.284251 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:32.284227    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 20:40:32.592626 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:32.592537    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mjrrg/must-gather-9scff" event={"ID":"3cddeb5e-ea42-4f71-b02a-64540f746a8e","Type":"ContainerStarted","Data":"927d698b90f7fcb142a26fe1d6af5075fcab9d68f65f3b7a74d80bf7c1cead2d"}
Sep 11 20:40:33.603057 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:33.603018    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mjrrg/must-gather-9scff" event={"ID":"3cddeb5e-ea42-4f71-b02a-64540f746a8e","Type":"ContainerStarted","Data":"8841aa80d5063d3e4b7c45684c7c3b62785bf4d90171f4e0f77ef15a807c73d8"}
Sep 11 20:40:33.603441 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:33.603067    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mjrrg/must-gather-9scff" event={"ID":"3cddeb5e-ea42-4f71-b02a-64540f746a8e","Type":"ContainerStarted","Data":"c6afa643167b977f6ebb210862a9de763af403af1d831f689d9a37f55384f6d3"}
Sep 11 20:40:33.618353 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:33.618294    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mjrrg/must-gather-9scff" podStartSLOduration=1.876182837 podStartE2EDuration="2.618271599s" totalImagesPullingTime="742.088762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:40:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:40:33.616749984 +0000 UTC m=+2403.608912283" watchObservedRunningTime="2026-09-11 20:40:33.618271599 +0000 UTC m=+2403.610433870"
Sep 11 20:40:34.441597 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:34.441562    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-95wcq_2936adde-ce1b-4859-858d-db3584a92609/global-pull-secret-syncer/0.log"
Sep 11 20:40:34.581044 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:34.581015    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vg4js_0ddb4097-dd15-4361-8302-fded613f0568/konnectivity-agent/0.log"
Sep 11 20:40:34.624195 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:34.624115    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-209.ec2.internal_abeccc9e8a8aea0630d3515e20a0e542/haproxy/0.log"
Sep 11 20:40:39.437515 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:39.436626    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b84cff79-88hf6_a1f9e71e-d948-4142-a345-5e8231aabf2d/authorino/0.log"
Sep 11 20:40:39.497167 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:39.497135    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-tx4dp_2f73cdfa-3681-4c39-915b-366711653f6f/manager/0.log"
Sep 11 20:40:39.517320 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:39.517287    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-5dnxt_e6bd712e-5be3-4bee-8974-76dff2193cac/kuadrant-console-plugin/0.log"
Sep 11 20:40:39.573080 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:39.573043    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-7kjmw_6f53db6b-18c9-46a8-801a-da9c44c8bc17/limitador/0.log"
Sep 11 20:40:41.328142 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:41.328102    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-4n8lq_3146fb02-8e77-4589-b856-eac2cecf5326/kube-state-metrics/0.log"
Sep 11 20:40:41.347657 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:41.347624    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-4n8lq_3146fb02-8e77-4589-b856-eac2cecf5326/kube-rbac-proxy-main/0.log"
Sep 11 20:40:41.367050 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:41.367018    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-4n8lq_3146fb02-8e77-4589-b856-eac2cecf5326/kube-rbac-proxy-self/0.log"
Sep 11 20:40:41.390295 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:41.390269    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-954bf79db-prgff_98c6ffec-27b5-43c0-84d5-7884d6cb7099/metrics-server/0.log"
Sep 11 20:40:41.411383 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:41.411296    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-rnd5p_7e42b56b-595f-476f-8a6d-a407bbc00233/monitoring-plugin/0.log"
Sep 11 20:40:41.575620 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:41.575596    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lw7xn_8dad13e0-28a0-4906-bc88-53f2e6c8f298/node-exporter/0.log"
Sep 11 20:40:41.591816 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:41.591692    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lw7xn_8dad13e0-28a0-4906-bc88-53f2e6c8f298/kube-rbac-proxy/0.log"
Sep 11 20:40:41.608327 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:41.608301    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lw7xn_8dad13e0-28a0-4906-bc88-53f2e6c8f298/init-textfile/0.log"
Sep 11 20:40:41.632865 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:41.632839    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-tcl2z_28df1910-4b5d-486d-b41e-faad23a61df6/kube-rbac-proxy-main/0.log"
Sep 11 20:40:41.651068 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:41.650996    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-tcl2z_28df1910-4b5d-486d-b41e-faad23a61df6/kube-rbac-proxy-self/0.log"
Sep 11 20:40:41.669706 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:41.669681    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-tcl2z_28df1910-4b5d-486d-b41e-faad23a61df6/openshift-state-metrics/0.log"
Sep 11 20:40:44.469685 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:44.469645    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5"]
Sep 11 20:40:44.473944 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:44.473917    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5"
Sep 11 20:40:44.482456 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:44.482426    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5"]
Sep 11 20:40:44.571511 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:44.571325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c90a130-2a25-4d89-9b0e-317abd9331ea-lib-modules\") pod \"perf-node-gather-daemonset-q52k5\" (UID: \"1c90a130-2a25-4d89-9b0e-317abd9331ea\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5"
Sep 11 20:40:44.571511 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:44.571419    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1c90a130-2a25-4d89-9b0e-317abd9331ea-podres\") pod \"perf-node-gather-daemonset-q52k5\" (UID: \"1c90a130-2a25-4d89-9b0e-317abd9331ea\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5"
Sep 11 20:40:44.571511 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:44.571485    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1c90a130-2a25-4d89-9b0e-317abd9331ea-proc\") pod \"perf-node-gather-daemonset-q52k5\" (UID: \"1c90a130-2a25-4d89-9b0e-317abd9331ea\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5"
Sep 11 20:40:44.571833 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:44.571563    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c90a130-2a25-4d89-9b0e-317abd9331ea-sys\") pod \"perf-node-gather-daemonset-q52k5\" (UID: \"1c90a130-2a25-4d89-9b0e-317abd9331ea\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5"
Sep 11 20:40:44.571833 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:44.571601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xttd9\" (UniqueName: \"kubernetes.io/projected/1c90a130-2a25-4d89-9b0e-317abd9331ea-kube-api-access-xttd9\") pod \"perf-node-gather-daemonset-q52k5\" (UID: \"1c90a130-2a25-4d89-9b0e-317abd9331ea\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5"
Sep 11 20:40:44.672804 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:44.672765    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1c90a130-2a25-4d89-9b0e-317abd9331ea-podres\") pod \"perf-node-gather-daemonset-q52k5\" (UID: \"1c90a130-2a25-4d89-9b0e-317abd9331ea\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5"
Sep 11 20:40:44.673091 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:44.673072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1c90a130-2a25-4d89-9b0e-317abd9331ea-proc\") pod \"perf-node-gather-daemonset-q52k5\" (UID: \"1c90a130-2a25-4d89-9b0e-317abd9331ea\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5"
Sep 11 20:40:44.673250 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:44.673234    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c90a130-2a25-4d89-9b0e-317abd9331ea-sys\") pod \"perf-node-gather-daemonset-q52k5\" (UID: \"1c90a130-2a25-4d89-9b0e-317abd9331ea\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5"
Sep 11 20:40:44.673368 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:44.673353    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xttd9\" (UniqueName: \"kubernetes.io/projected/1c90a130-2a25-4d89-9b0e-317abd9331ea-kube-api-access-xttd9\") pod \"perf-node-gather-daemonset-q52k5\" (UID: \"1c90a130-2a25-4d89-9b0e-317abd9331ea\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5"
Sep 11 20:40:44.673479 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:44.673466    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c90a130-2a25-4d89-9b0e-317abd9331ea-lib-modules\") pod \"perf-node-gather-daemonset-q52k5\" (UID: \"1c90a130-2a25-4d89-9b0e-317abd9331ea\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5"
Sep 11 20:40:44.673712 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:44.673692    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c90a130-2a25-4d89-9b0e-317abd9331ea-sys\") pod \"perf-node-gather-daemonset-q52k5\" (UID: \"1c90a130-2a25-4d89-9b0e-317abd9331ea\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5"
Sep 11 20:40:44.673821 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:44.673779    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1c90a130-2a25-4d89-9b0e-317abd9331ea-podres\") pod \"perf-node-gather-daemonset-q52k5\" (UID: \"1c90a130-2a25-4d89-9b0e-317abd9331ea\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5"
Sep 11 20:40:44.673901 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:44.673827    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1c90a130-2a25-4d89-9b0e-317abd9331ea-proc\") pod \"perf-node-gather-daemonset-q52k5\" (UID: \"1c90a130-2a25-4d89-9b0e-317abd9331ea\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5"
Sep 11 20:40:44.673991 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:44.673916    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c90a130-2a25-4d89-9b0e-317abd9331ea-lib-modules\") pod \"perf-node-gather-daemonset-q52k5\" (UID: \"1c90a130-2a25-4d89-9b0e-317abd9331ea\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5"
Sep 11 20:40:44.685281 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:44.685177    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xttd9\" (UniqueName: \"kubernetes.io/projected/1c90a130-2a25-4d89-9b0e-317abd9331ea-kube-api-access-xttd9\") pod \"perf-node-gather-daemonset-q52k5\" (UID: \"1c90a130-2a25-4d89-9b0e-317abd9331ea\") " pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5"
Sep 11 20:40:44.793300 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:44.793215    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5"
Sep 11 20:40:44.889781 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:44.889618    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-wzwg7_df290aa2-b262-4940-9eea-054aca381519/download-server/0.log"
Sep 11 20:40:45.089227 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:45.089193    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5"]
Sep 11 20:40:45.089939 ip-10-0-128-209 kubenswrapper[2580]: W0911 20:40:45.089889    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1c90a130_2a25_4d89_9b0e_317abd9331ea.slice/crio-da0ab89fc0a10abc49bbc79d4a3fc9e600ee3b7c8f9653a9bf21a4b26ff30313 WatchSource:0}: Error finding container da0ab89fc0a10abc49bbc79d4a3fc9e600ee3b7c8f9653a9bf21a4b26ff30313: Status 404 returned error can't find the container with id da0ab89fc0a10abc49bbc79d4a3fc9e600ee3b7c8f9653a9bf21a4b26ff30313
Sep 11 20:40:45.729318 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:45.729277    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5" event={"ID":"1c90a130-2a25-4d89-9b0e-317abd9331ea","Type":"ContainerStarted","Data":"e6f9fece53e477eb22e192b33af2362a8ac45d07267e4a7ab7d751c0a7225f6d"}
Sep 11 20:40:45.729845 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:45.729345    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5" event={"ID":"1c90a130-2a25-4d89-9b0e-317abd9331ea","Type":"ContainerStarted","Data":"da0ab89fc0a10abc49bbc79d4a3fc9e600ee3b7c8f9653a9bf21a4b26ff30313"}
Sep 11 20:40:45.730158 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:45.730132    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5"
Sep 11 20:40:45.746358 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:45.746294    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5" podStartSLOduration=1.746274535 podStartE2EDuration="1.746274535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:40:45.74455895 +0000 UTC m=+2415.736721221" watchObservedRunningTime="2026-09-11 20:40:45.746274535 +0000 UTC m=+2415.738436807"
Sep 11 20:40:46.605871 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:46.605844    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q484p_a9498946-2c6a-4a09-ae75-a79812f06fd2/dns/0.log"
Sep 11 20:40:46.621678 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:46.621653    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q484p_a9498946-2c6a-4a09-ae75-a79812f06fd2/kube-rbac-proxy/0.log"
Sep 11 20:40:46.705730 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:46.705702    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j9pns_83199f92-0393-4012-96cb-bb5d3172786f/dns-node-resolver/0.log"
Sep 11 20:40:47.215564 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:47.215537    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-68887c9c9b-thsrj_1819bd16-803f-4ec5-a141-5911232d0799/registry/0.log"
Sep 11 20:40:47.252983 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:47.252951    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-knjv8_59344c06-c093-43c1-8ef5-9e3df17dd1cd/node-ca/0.log"
Sep 11 20:40:48.556277 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:48.556203    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-567ff9dcd5-hrxd6_956f104b-830b-4d30-8185-d446debf54c3/kube-auth-proxy/0.log"
Sep 11 20:40:48.665443 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:48.665413    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jtmt6_76f2ec85-7849-4143-9b55-65fe0133f662/payload-processing/0.log"
Sep 11 20:40:49.548858 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:49.548830    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rssdb_def84bb5-29ef-44af-a9b8-34a1f3715012/serve-healthcheck-canary/0.log"
Sep 11 20:40:50.011762 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:50.011724    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cchrc_2197b3ff-694b-4b17-9916-149ec1eb9924/kube-rbac-proxy/0.log"
Sep 11 20:40:50.027914 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:50.027879    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cchrc_2197b3ff-694b-4b17-9916-149ec1eb9924/exporter/0.log"
Sep 11 20:40:50.044315 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:50.044289    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cchrc_2197b3ff-694b-4b17-9916-149ec1eb9924/extractor/0.log"
Sep 11 20:40:52.699851 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:52.699816    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fdf48ff49-24xlt_4bb46042-91b3-4506-857f-4db20fa98890/manager/0.log"
Sep 11 20:40:52.715678 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:52.715654    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fdf48ff49-24xlt_4bb46042-91b3-4506-857f-4db20fa98890/copy-manifests/0.log"
Sep 11 20:40:52.753883 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:52.753851    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mjrrg/perf-node-gather-daemonset-q52k5"
Sep 11 20:40:54.112400 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:54.112371    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-865445d586-svn2l_638ffad9-e877-4c31-8975-aeede2e0b352/manager/0.log"
Sep 11 20:40:54.724084 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:54.724051    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-567ff9dcd5-hrxd6_956f104b-830b-4d30-8185-d446debf54c3/kube-auth-proxy/0.log"
Sep 11 20:40:54.816819 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:54.816726    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jtmt6_76f2ec85-7849-4143-9b55-65fe0133f662/payload-processing/0.log"
Sep 11 20:40:56.811448 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:56.811416    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-567ff9dcd5-hrxd6_956f104b-830b-4d30-8185-d446debf54c3/kube-auth-proxy/0.log"
Sep 11 20:40:56.895664 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:40:56.895631    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-jtmt6_76f2ec85-7849-4143-9b55-65fe0133f662/payload-processing/0.log"
Sep 11 20:41:00.599035 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:41:00.598999    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8nw2z_562cad6a-184c-4433-be65-0cd67d1a29d2/kube-multus/0.log"
Sep 11 20:41:00.621829 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:41:00.621802    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9n8ls_4ceeee54-3eb9-4248-87ff-63b7da3896ce/kube-multus-additional-cni-plugins/0.log"
Sep 11 20:41:00.639853 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:41:00.639831    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9n8ls_4ceeee54-3eb9-4248-87ff-63b7da3896ce/egress-router-binary-copy/0.log"
Sep 11 20:41:00.656216 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:41:00.656195    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9n8ls_4ceeee54-3eb9-4248-87ff-63b7da3896ce/cni-plugins/0.log"
Sep 11 20:41:00.671454 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:41:00.671434    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9n8ls_4ceeee54-3eb9-4248-87ff-63b7da3896ce/bond-cni-plugin/0.log"
Sep 11 20:41:00.689643 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:41:00.689628    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9n8ls_4ceeee54-3eb9-4248-87ff-63b7da3896ce/routeoverride-cni/0.log"
Sep 11 20:41:00.704936 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:41:00.704898    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9n8ls_4ceeee54-3eb9-4248-87ff-63b7da3896ce/whereabouts-cni-bincopy/0.log"
Sep 11 20:41:00.720966 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:41:00.720948    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9n8ls_4ceeee54-3eb9-4248-87ff-63b7da3896ce/whereabouts-cni/0.log"
Sep 11 20:41:01.026278 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:41:01.026244    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fp6qw_dda12783-5f3e-4a3e-9fde-4681797a7b60/network-metrics-daemon/0.log"
Sep 11 20:41:01.037921 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:41:01.037891    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fp6qw_dda12783-5f3e-4a3e-9fde-4681797a7b60/kube-rbac-proxy/0.log"
Sep 11 20:41:01.851960 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:41:01.851927    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7pjf_5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6/ovn-controller/0.log"
Sep 11 20:41:01.864845 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:41:01.864815    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7pjf_5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6/ovn-acl-logging/0.log"
Sep 11 20:41:01.883956 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:41:01.883936    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7pjf_5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6/kube-rbac-proxy-node/0.log"
Sep 11 20:41:01.899704 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:41:01.899682    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7pjf_5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6/kube-rbac-proxy-ovn-metrics/0.log"
Sep 11 20:41:01.914195 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:41:01.914178    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7pjf_5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6/northd/0.log"
Sep 11 20:41:01.932373 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:41:01.932349    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7pjf_5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6/nbdb/0.log"
Sep 11 20:41:01.950410 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:41:01.950387    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7pjf_5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6/sbdb/0.log"
Sep 11 20:41:02.014643 ip-10-0-128-209 kubenswrapper[2580]: I0911 20:41:02.014613    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-j7pjf_5c0dbfa7-f635-4fa5-9c1a-5cd4dcd211e6/ovnkube-controller/0.log"