Sep 05 19:43:19.687806 ip-10-0-142-208 systemd[1]: Starting Kubernetes Kubelet...
Sep 05 19:43:20.072737 ip-10-0-142-208 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 19:43:20.072737 ip-10-0-142-208 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 05 19:43:20.072737 ip-10-0-142-208 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 19:43:20.072737 ip-10-0-142-208 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 05 19:43:20.072737 ip-10-0-142-208 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 05 19:43:20.075320 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.075228    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 05 19:43:20.080185 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080169    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 19:43:20.080185 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080186    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 19:43:20.080258 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080190    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 19:43:20.080258 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080194    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 19:43:20.080258 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080197    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 19:43:20.080258 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080200    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 19:43:20.080258 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080203    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 19:43:20.080258 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080206    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 19:43:20.080258 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080209    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 19:43:20.080258 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080211    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 19:43:20.080258 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080214    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 19:43:20.080258 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080217    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 19:43:20.080258 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080219    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 19:43:20.080258 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080222    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 19:43:20.080258 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080224    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 19:43:20.080258 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080227    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 19:43:20.080258 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080229    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 19:43:20.080258 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080232    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 19:43:20.080258 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080234    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 19:43:20.080258 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080237    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 19:43:20.080258 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080240    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 19:43:20.080726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080244    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 19:43:20.080726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080246    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 19:43:20.080726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080249    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 19:43:20.080726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080251    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 19:43:20.080726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080254    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 19:43:20.080726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080258    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 19:43:20.080726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080262    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 19:43:20.080726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080265    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 19:43:20.080726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080268    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 19:43:20.080726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080271    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 19:43:20.080726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080274    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 19:43:20.080726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080277    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 19:43:20.080726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080279    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 19:43:20.080726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080282    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 19:43:20.080726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080284    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 19:43:20.080726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080288    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 19:43:20.080726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080291    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 19:43:20.080726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080293    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 19:43:20.080726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080296    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 19:43:20.080726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080299    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 19:43:20.081219 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080301    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 19:43:20.081219 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080303    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 19:43:20.081219 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080306    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 19:43:20.081219 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080310    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 19:43:20.081219 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080314    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 19:43:20.081219 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080316    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 19:43:20.081219 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080319    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 19:43:20.081219 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080322    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 19:43:20.081219 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080325    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 19:43:20.081219 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080328    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 19:43:20.081219 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080330    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 19:43:20.081219 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080333    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 19:43:20.081219 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080336    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 19:43:20.081219 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080340    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 19:43:20.081219 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080342    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 19:43:20.081219 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080345    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 19:43:20.081219 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080348    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 19:43:20.081219 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080350    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 19:43:20.081219 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080353    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 19:43:20.081219 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080355    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 19:43:20.081734 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080358    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 19:43:20.081734 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080360    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 19:43:20.081734 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080363    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 19:43:20.081734 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080365    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 19:43:20.081734 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080367    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 19:43:20.081734 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080370    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 19:43:20.081734 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080372    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 19:43:20.081734 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080375    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 19:43:20.081734 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080378    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 19:43:20.081734 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080380    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 19:43:20.081734 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080383    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 19:43:20.081734 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080386    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 19:43:20.081734 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080388    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 19:43:20.081734 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080390    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 19:43:20.081734 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080393    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 19:43:20.081734 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080396    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 19:43:20.081734 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080399    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 19:43:20.081734 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080401    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 19:43:20.081734 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080404    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 19:43:20.081734 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080406    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 19:43:20.082224 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080408    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 19:43:20.082224 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080411    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 19:43:20.082224 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080414    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 19:43:20.082224 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080416    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 19:43:20.082224 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080418    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 19:43:20.082224 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080423    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 19:43:20.082224 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080427    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 19:43:20.082224 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080431    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 19:43:20.082224 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080434    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 19:43:20.082224 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080437    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 19:43:20.082224 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080440    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 19:43:20.082224 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080442    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 19:43:20.082224 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080445    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 19:43:20.082224 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080448    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 19:43:20.082224 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080450    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 19:43:20.082224 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080453    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 19:43:20.082224 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080456    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 19:43:20.082224 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.080458    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 19:43:20.082224 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081200    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 19:43:20.082224 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081208    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 19:43:20.082726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081212    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 19:43:20.082726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081216    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 19:43:20.082726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081219    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 19:43:20.082726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081222    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 19:43:20.082726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081226    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 19:43:20.082726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081229    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 19:43:20.082726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081232    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 19:43:20.082726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081235    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 19:43:20.082726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081238    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 19:43:20.082726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081240    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 19:43:20.082726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081243    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 19:43:20.082726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081246    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 19:43:20.082726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081248    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 19:43:20.082726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081251    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 19:43:20.082726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081254    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 19:43:20.082726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081257    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 19:43:20.082726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081260    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 19:43:20.082726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081264    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 19:43:20.082726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081267    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 19:43:20.083201 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081270    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 19:43:20.083201 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081272    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 19:43:20.083201 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081275    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 19:43:20.083201 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081278    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 19:43:20.083201 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081280    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 19:43:20.083201 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081283    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 19:43:20.083201 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081285    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 19:43:20.083201 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081288    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 19:43:20.083201 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081290    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 19:43:20.083201 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081293    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 19:43:20.083201 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081295    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 19:43:20.083201 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081298    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 19:43:20.083201 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081300    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 19:43:20.083201 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081303    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 19:43:20.083201 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081306    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 19:43:20.083201 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081308    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 19:43:20.083201 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081311    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 19:43:20.083201 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081313    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 19:43:20.083201 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081316    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 19:43:20.083201 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081319    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 19:43:20.083704 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081321    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 19:43:20.083704 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081324    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 19:43:20.083704 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081326    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 19:43:20.083704 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081329    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 19:43:20.083704 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081331    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 19:43:20.083704 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081334    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 19:43:20.083704 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081338    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 19:43:20.083704 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081341    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 19:43:20.083704 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081344    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 19:43:20.083704 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081346    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 19:43:20.083704 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081350    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 19:43:20.083704 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081352    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 19:43:20.083704 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081355    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 19:43:20.083704 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081359    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 19:43:20.083704 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081361    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 19:43:20.083704 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081365    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 19:43:20.083704 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081369    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 19:43:20.083704 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081372    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 19:43:20.083704 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081375    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 19:43:20.083704 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081378    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 19:43:20.084176 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081380    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 19:43:20.084176 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081383    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 19:43:20.084176 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081386    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 19:43:20.084176 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081388    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 19:43:20.084176 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081391    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 19:43:20.084176 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081394    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 19:43:20.084176 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081396    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 19:43:20.084176 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081399    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 19:43:20.084176 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081402    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 19:43:20.084176 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081404    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 19:43:20.084176 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081407    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 19:43:20.084176 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081410    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 19:43:20.084176 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081412    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 19:43:20.084176 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081415    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 19:43:20.084176 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081417    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 19:43:20.084176 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081420    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 19:43:20.084176 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081423    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 19:43:20.084176 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081427    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 19:43:20.084176 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081431    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 19:43:20.084655 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081434    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 19:43:20.084655 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081437    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 19:43:20.084655 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081440    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 19:43:20.084655 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081443    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 19:43:20.084655 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081445    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 19:43:20.084655 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081448    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 19:43:20.084655 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081451    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 19:43:20.084655 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081454    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 19:43:20.084655 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081458    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 19:43:20.084655 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081461    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 19:43:20.084655 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081464    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 19:43:20.084655 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081466    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 19:43:20.084655 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081469    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 19:43:20.084655 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081473    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 19:43:20.084655 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081475    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 19:43:20.084655 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081478    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 19:43:20.084655 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081481    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 19:43:20.084655 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081484    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 19:43:20.084655 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081487    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 19:43:20.085114 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081893    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 19:43:20.085114 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081899    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 19:43:20.085114 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081902    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 19:43:20.085114 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081905    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 19:43:20.085114 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081908    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 19:43:20.085114 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081911    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 19:43:20.085114 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081915    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 19:43:20.085114 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081919    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 19:43:20.085114 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081923    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 19:43:20.085114 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081926    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 19:43:20.085114 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081928    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 19:43:20.085114 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081931    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 19:43:20.085114 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081934    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 19:43:20.085114 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081937    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 19:43:20.085114 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081941    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 19:43:20.085114 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081944    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 19:43:20.085114 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081947    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 19:43:20.085114 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081951    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 19:43:20.085114 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081955    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 19:43:20.085614 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081958    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 19:43:20.085614 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081961    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 19:43:20.085614 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081964    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 19:43:20.085614 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081967    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 19:43:20.085614 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081969    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 19:43:20.085614 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081972    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 19:43:20.085614 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081974    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 19:43:20.085614 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081977    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 19:43:20.085614 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081980    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 19:43:20.085614 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081983    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 19:43:20.085614 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081985    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 19:43:20.085614 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081988    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 19:43:20.085614 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081990    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 19:43:20.085614 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081995    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 19:43:20.085614 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.081998    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 19:43:20.085614 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082001    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 19:43:20.085614 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082004    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 19:43:20.085614 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082007    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 19:43:20.085614 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082009    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 19:43:20.085614 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082012    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 19:43:20.086102 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082015    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 19:43:20.086102 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082017    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 19:43:20.086102 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082020    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 19:43:20.086102 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082022    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 19:43:20.086102 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082025    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 19:43:20.086102 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082027    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 19:43:20.086102 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082030    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 19:43:20.086102 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082033    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 19:43:20.086102 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082035    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 19:43:20.086102 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082038    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 19:43:20.086102 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082040    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 19:43:20.086102 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082043    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 19:43:20.086102 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082046    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 19:43:20.086102 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082048    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 19:43:20.086102 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082051    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 19:43:20.086102 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082054    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 19:43:20.086102 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082056    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 19:43:20.086102 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082059    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 19:43:20.086102 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082062    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 19:43:20.086102 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082064    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 19:43:20.086665 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082067    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 19:43:20.086665 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082070    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 19:43:20.086665 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082072    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 19:43:20.086665 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082075    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 19:43:20.086665 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082078    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 19:43:20.086665 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082081    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 19:43:20.086665 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082084    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 19:43:20.086665 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082087    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 19:43:20.086665 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082089    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 19:43:20.086665 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082092    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 19:43:20.086665 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082095    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 19:43:20.086665 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082097    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 19:43:20.086665 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082100    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 19:43:20.086665 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082102    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 19:43:20.086665 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082106    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 19:43:20.086665 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082108    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 19:43:20.086665 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082111    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 19:43:20.086665 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082114    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 19:43:20.086665 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082116    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 19:43:20.086665 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082119    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 19:43:20.087156 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082122    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 19:43:20.087156 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082125    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 19:43:20.087156 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082128    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 19:43:20.087156 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082131    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 19:43:20.087156 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082133    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 19:43:20.087156 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082136    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 19:43:20.087156 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082139    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 19:43:20.087156 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082142    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 19:43:20.087156 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082144    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 19:43:20.087156 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082148    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 19:43:20.087156 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082152    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 19:43:20.087156 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082155    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 19:43:20.087156 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082158    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 19:43:20.087156 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082161    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 19:43:20.087156 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082176    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 19:43:20.087156 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082179    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 19:43:20.087156 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082182    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 19:43:20.087156 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082186    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 19:43:20.087156 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082189    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 19:43:20.087156 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082191    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 19:43:20.087664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082256    2583 flags.go:64] FLAG: --address="0.0.0.0"
Sep 05 19:43:20.087664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082296    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 05 19:43:20.087664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082302    2583 flags.go:64] FLAG: --anonymous-auth="true"
Sep 05 19:43:20.087664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082307    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 05 19:43:20.087664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082311    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 05 19:43:20.087664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082314    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 05 19:43:20.087664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082318    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 05 19:43:20.087664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082323    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 05 19:43:20.087664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082326    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 05 19:43:20.087664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082329    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 05 19:43:20.087664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082333    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 05 19:43:20.087664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082336    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 05 19:43:20.087664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082339    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 05 19:43:20.087664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082342    2583 flags.go:64] FLAG: --cgroup-root=""
Sep 05 19:43:20.087664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082346    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 05 19:43:20.087664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082348    2583 flags.go:64] FLAG: --client-ca-file=""
Sep 05 19:43:20.087664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082351    2583 flags.go:64] FLAG: --cloud-provider="external"
Sep 05 19:43:20.087664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082354    2583 flags.go:64] FLAG: --cluster-dns="[]"
Sep 05 19:43:20.087664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082359    2583 flags.go:64] FLAG: --cluster-domain=""
Sep 05 19:43:20.087664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082362    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 05 19:43:20.087664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082365    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 05 19:43:20.087664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082368    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 05 19:43:20.087664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082372    2583 flags.go:64] FLAG: --container-log-max-files="5"
Sep 05 19:43:20.087664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082375    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082378    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082381    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082385    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082388    2583 flags.go:64] FLAG: --contention-profiling="false"
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082391    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082395    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082398    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082401    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082405    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082408    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082411    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082415    2583 flags.go:64] FLAG: --enable-load-reader="false"
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082418    2583 flags.go:64] FLAG: --enable-server="true"
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082421    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082425    2583 flags.go:64] FLAG: --event-burst="100"
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082428    2583 flags.go:64] FLAG: --event-qps="50"
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082431    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082434    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082437    2583 flags.go:64] FLAG: --eviction-hard=""
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082442    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082445    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082448    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082451    2583 flags.go:64] FLAG: --eviction-soft=""
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082454    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 05 19:43:20.088286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082457    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 05 19:43:20.088916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082460    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 05 19:43:20.088916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082463    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 05 19:43:20.088916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082466    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 05 19:43:20.088916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082469    2583 flags.go:64] FLAG: --fail-swap-on="true"
Sep 05 19:43:20.088916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082472    2583 flags.go:64] FLAG: --feature-gates=""
Sep 05 19:43:20.088916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082476    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 05 19:43:20.088916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082479    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 05 19:43:20.088916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082482    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 05 19:43:20.088916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082485    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 05 19:43:20.088916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082488    2583 flags.go:64] FLAG: --healthz-port="10248"
Sep 05 19:43:20.088916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082491    2583 flags.go:64] FLAG: --help="false"
Sep 05 19:43:20.088916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082494    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-142-208.ec2.internal"
Sep 05 19:43:20.088916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082499    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 05 19:43:20.088916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082502    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 05 19:43:20.088916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082504    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 05 19:43:20.088916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082508    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 05 19:43:20.088916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082511    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 05 19:43:20.088916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082528    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 05 19:43:20.088916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082532    2583 flags.go:64] FLAG: --image-service-endpoint=""
Sep 05 19:43:20.088916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082535    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 05 19:43:20.088916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082538    2583 flags.go:64] FLAG: --kube-api-burst="100"
Sep 05 19:43:20.088916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082541    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 05 19:43:20.088916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082544    2583 flags.go:64] FLAG: --kube-api-qps="50"
Sep 05 19:43:20.088916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082547    2583 flags.go:64] FLAG: --kube-reserved=""
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082550    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082553    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082558    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082560    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082563    2583 flags.go:64] FLAG: --lock-file=""
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082566    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082569    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082572    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082577    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082580    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082584    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082587    2583 flags.go:64] FLAG: --logging-format="text"
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082590    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082594    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082597    2583 flags.go:64] FLAG: --manifest-url=""
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082600    2583 flags.go:64] FLAG: --manifest-url-header=""
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082604    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082607    2583 flags.go:64] FLAG: --max-open-files="1000000"
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082611    2583 flags.go:64] FLAG: --max-pods="110"
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082614    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082619    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082622    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082625    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082628    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 05 19:43:20.089596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082631    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082634    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082641    2583 flags.go:64] FLAG: --node-status-max-images="50"
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082644    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082647    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082650    2583 flags.go:64] FLAG: --pod-cidr=""
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082653    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082658    2583 flags.go:64] FLAG: --pod-manifest-path=""
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082661    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082664    2583 flags.go:64] FLAG: --pods-per-core="0"
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082667    2583 flags.go:64] FLAG: --port="10250"
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082673    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082677    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0daeee354c6f25430"
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082680    2583 flags.go:64] FLAG: --qos-reserved=""
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082683    2583 flags.go:64] FLAG: --read-only-port="10255"
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082686    2583 flags.go:64] FLAG: --register-node="true"
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082689    2583 flags.go:64] FLAG: --register-with-taints=""
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082692    2583 flags.go:64] FLAG: --registry-burst="10"
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082695    2583 flags.go:64] FLAG: --registry-qps="5"
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082698    2583 flags.go:64] FLAG: --reserved-cpus=""
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082701    2583 flags.go:64] FLAG: --reserved-memory=""
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082705    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082708    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082711    2583 flags.go:64] FLAG: --rotate-certificates="false"
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082714    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 05 19:43:20.090425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082717    2583 flags.go:64] FLAG: --runonce="false"
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082720    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082723    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082728    2583 flags.go:64] FLAG: --seccomp-default="false"
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082731    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082734    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082737    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082740    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082743    2583 flags.go:64] FLAG: --storage-driver-password="root"
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082746    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082749    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082752    2583 flags.go:64] FLAG: --storage-driver-user="root"
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082755    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082758    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082761    2583 flags.go:64] FLAG: --system-cgroups=""
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082764    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082769    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082772    2583 flags.go:64] FLAG: --tls-cert-file=""
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082775    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082779    2583 flags.go:64] FLAG: --tls-min-version=""
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082782    2583 flags.go:64] FLAG: --tls-private-key-file=""
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082785    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082788    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082791    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082794    2583 flags.go:64] FLAG: --v="2"
Sep 05 19:43:20.091321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082798    2583 flags.go:64] FLAG: --version="false"
Sep 05 19:43:20.091968 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082801    2583 flags.go:64] FLAG: --vmodule=""
Sep 05 19:43:20.091968 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082805    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 05 19:43:20.091968 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.082808    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 05 19:43:20.091968 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082922    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 19:43:20.091968 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082925    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 19:43:20.091968 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082928    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 19:43:20.091968 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082931    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 19:43:20.091968 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082935    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 19:43:20.091968 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082938    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 19:43:20.091968 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082942    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 19:43:20.091968 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082945    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 19:43:20.091968 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082948    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 19:43:20.091968 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082951    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 19:43:20.091968 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082953    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 19:43:20.091968 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082956    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 19:43:20.091968 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082960    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 19:43:20.091968 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082962    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 19:43:20.091968 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082965    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 19:43:20.091968 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082968    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 19:43:20.091968 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082971    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 19:43:20.092496 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082974    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 19:43:20.092496 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082977    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 19:43:20.092496 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082979    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 19:43:20.092496 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082982    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 19:43:20.092496 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082986    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 19:43:20.092496 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082988    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 19:43:20.092496 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082991    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 19:43:20.092496 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082993    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 19:43:20.092496 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082996    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 19:43:20.092496 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.082998    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 19:43:20.092496 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083001    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 19:43:20.092496 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083004    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 19:43:20.092496 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083006    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 19:43:20.092496 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083009    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 19:43:20.092496 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083011    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 19:43:20.092496 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083014    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 19:43:20.092496 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083017    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 19:43:20.092496 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083021    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 19:43:20.092496 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083024    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 19:43:20.092972 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083027    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 19:43:20.092972 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083030    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 19:43:20.092972 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083033    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 19:43:20.092972 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083035    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 19:43:20.092972 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083039    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 19:43:20.092972 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083041    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 19:43:20.092972 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083044    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 19:43:20.092972 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083046    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 19:43:20.092972 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083048    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 19:43:20.092972 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083051    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 19:43:20.092972 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083053    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 19:43:20.092972 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083056    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 19:43:20.092972 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083058    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 19:43:20.092972 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083061    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 19:43:20.092972 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083063    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 19:43:20.092972 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083066    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 19:43:20.092972 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083068    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 19:43:20.092972 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083072    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 19:43:20.092972 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083075    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 19:43:20.092972 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083077    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 19:43:20.093452 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083080    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 19:43:20.093452 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083083    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 19:43:20.093452 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083085    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 19:43:20.093452 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083088    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 19:43:20.093452 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083090    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 19:43:20.093452 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083093    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 19:43:20.093452 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083095    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 19:43:20.093452 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083098    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 19:43:20.093452 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083100    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 19:43:20.093452 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083103    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 19:43:20.093452 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083106    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 19:43:20.093452 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083108    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 19:43:20.093452 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083111    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 19:43:20.093452 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083114    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 19:43:20.093452 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083118    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 19:43:20.093452 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083121    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 19:43:20.093452 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083123    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 19:43:20.093452 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083126    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 19:43:20.093452 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083128    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 19:43:20.093959 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083131    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 19:43:20.093959 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083133    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 19:43:20.093959 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083136    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 19:43:20.093959 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083138    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 19:43:20.093959 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083141    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 19:43:20.093959 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083143    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 19:43:20.093959 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083146    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 19:43:20.093959 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083148    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 19:43:20.093959 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083150    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 19:43:20.093959 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083153    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 19:43:20.093959 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083156    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 19:43:20.093959 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083159    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 19:43:20.093959 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083161    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 19:43:20.093959 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083164    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 19:43:20.093959 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083167    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 19:43:20.093959 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083169    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 19:43:20.093959 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083172    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 19:43:20.093959 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083174    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 19:43:20.093959 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083177    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 19:43:20.093959 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083179    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 19:43:20.094471 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083182    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 19:43:20.094471 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083184    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 19:43:20.094471 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083187    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 19:43:20.094471 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.083189    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 19:43:20.094471 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.083832    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 19:43:20.094471 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.092408    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 05 19:43:20.094471 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.092425    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 05 19:43:20.094471 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.092444    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 05 19:43:20.094471 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.092452    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 05 19:43:20.094471 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092512    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 19:43:20.094471 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092534    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 19:43:20.094471 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092538    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 19:43:20.094471 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092541    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 19:43:20.094471 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092544    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 19:43:20.094471 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092547    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 19:43:20.094865 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092550    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 19:43:20.094865 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092553    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 19:43:20.094865 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092556    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 19:43:20.094865 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092559    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 19:43:20.094865 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092562    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 19:43:20.094865 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092565    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 19:43:20.094865 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092568    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 19:43:20.094865 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092571    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 19:43:20.094865 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092575    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 19:43:20.094865 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092578    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 19:43:20.094865 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092581    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 19:43:20.094865 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092585    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 19:43:20.094865 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092590    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 19:43:20.094865 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092593    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 19:43:20.094865 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092598    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 19:43:20.094865 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092603    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 19:43:20.094865 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092608    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 19:43:20.094865 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092612    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 19:43:20.094865 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092615    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 19:43:20.095331 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092618    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 19:43:20.095331 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092621    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 19:43:20.095331 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092624    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 19:43:20.095331 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092627    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 19:43:20.095331 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092631    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 19:43:20.095331 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092634    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 19:43:20.095331 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092637    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 19:43:20.095331 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092640    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 19:43:20.095331 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092643    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 19:43:20.095331 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092645    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 19:43:20.095331 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092648    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 19:43:20.095331 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092650    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 19:43:20.095331 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092653    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 19:43:20.095331 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092655    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 19:43:20.095331 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092658    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 19:43:20.095331 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092660    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 19:43:20.095331 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092663    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 19:43:20.095331 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092666    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 19:43:20.095331 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092668    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 19:43:20.095331 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092671    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 19:43:20.095918 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092674    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 19:43:20.095918 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092680    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 19:43:20.095918 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092685    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 19:43:20.095918 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092688    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 19:43:20.095918 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092691    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 19:43:20.095918 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092693    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 19:43:20.095918 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092696    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 19:43:20.095918 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092699    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 19:43:20.095918 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092702    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 19:43:20.095918 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092706    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 19:43:20.095918 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092710    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 19:43:20.095918 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092712    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 19:43:20.095918 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092715    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 19:43:20.095918 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092718    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 19:43:20.095918 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092720    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 19:43:20.095918 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092723    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 19:43:20.095918 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092726    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 19:43:20.095918 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092729    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 19:43:20.095918 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092732    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 19:43:20.096383 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092734    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 19:43:20.096383 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092737    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 19:43:20.096383 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092740    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 19:43:20.096383 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092742    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 19:43:20.096383 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092745    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 19:43:20.096383 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092747    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 19:43:20.096383 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092750    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 19:43:20.096383 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092754    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 19:43:20.096383 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092758    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 19:43:20.096383 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092762    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 19:43:20.096383 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092765    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 19:43:20.096383 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092768    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 19:43:20.096383 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092770    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 19:43:20.096383 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092773    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 19:43:20.096383 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092776    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 19:43:20.096383 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092780    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 19:43:20.096383 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092782    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 19:43:20.096383 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092785    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 19:43:20.096383 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092787    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 19:43:20.096383 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092790    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 19:43:20.096879 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092792    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 19:43:20.096879 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092795    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 19:43:20.096879 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092797    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 19:43:20.096879 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092800    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 19:43:20.096879 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092802    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 19:43:20.096879 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092805    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 19:43:20.096879 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092807    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 19:43:20.096879 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092810    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 19:43:20.096879 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092813    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 19:43:20.096879 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092816    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 19:43:20.096879 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092819    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 19:43:20.096879 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092821    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 19:43:20.096879 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092824    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 19:43:20.096879 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092828    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 19:43:20.096879 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092832    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 19:43:20.097261 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.092840    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 19:43:20.097261 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092962    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 05 19:43:20.097261 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092968    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 05 19:43:20.097261 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092972    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 05 19:43:20.097261 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092975    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 05 19:43:20.097261 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092978    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 05 19:43:20.097261 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092982    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 05 19:43:20.097261 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092987    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 05 19:43:20.097261 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092993    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 05 19:43:20.097261 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.092999    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 05 19:43:20.097261 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093002    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 05 19:43:20.097261 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093005    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 05 19:43:20.097261 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093008    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 05 19:43:20.097261 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093012    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 05 19:43:20.097612 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093015    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 05 19:43:20.097612 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093018    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 05 19:43:20.097612 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093021    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 05 19:43:20.097612 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093024    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 05 19:43:20.097612 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093026    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 05 19:43:20.097612 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093029    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 05 19:43:20.097612 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093031    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 05 19:43:20.097612 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093034    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 05 19:43:20.097612 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093036    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 05 19:43:20.097612 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093039    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 05 19:43:20.097612 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093042    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 05 19:43:20.097612 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093044    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 05 19:43:20.097612 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093047    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 05 19:43:20.097612 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093050    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 05 19:43:20.097612 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093052    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 05 19:43:20.097612 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093055    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 05 19:43:20.097612 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093078    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 05 19:43:20.097612 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093082    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 05 19:43:20.097612 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093086    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 05 19:43:20.097612 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093089    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 05 19:43:20.098094 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093092    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 05 19:43:20.098094 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093096    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 05 19:43:20.098094 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093099    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 05 19:43:20.098094 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093102    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 05 19:43:20.098094 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093105    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 05 19:43:20.098094 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093107    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 05 19:43:20.098094 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093110    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 05 19:43:20.098094 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093112    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 05 19:43:20.098094 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093115    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 05 19:43:20.098094 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093118    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 05 19:43:20.098094 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093121    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 05 19:43:20.098094 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093123    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 05 19:43:20.098094 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093126    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 05 19:43:20.098094 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093129    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 05 19:43:20.098094 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093132    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 05 19:43:20.098094 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093134    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 05 19:43:20.098094 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093138    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 05 19:43:20.098094 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093142    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 05 19:43:20.098094 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093146    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 05 19:43:20.098591 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093150    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 05 19:43:20.098591 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093152    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 05 19:43:20.098591 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093155    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 05 19:43:20.098591 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093157    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 05 19:43:20.098591 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093160    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 05 19:43:20.098591 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093163    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 05 19:43:20.098591 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093165    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 05 19:43:20.098591 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093168    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 05 19:43:20.098591 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093170    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 05 19:43:20.098591 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093173    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 05 19:43:20.098591 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093175    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 05 19:43:20.098591 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093178    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 05 19:43:20.098591 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093181    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 05 19:43:20.098591 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093183    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 05 19:43:20.098591 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093186    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 05 19:43:20.098591 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093189    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 05 19:43:20.098591 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093191    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 05 19:43:20.098591 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093194    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 05 19:43:20.098591 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093196    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 05 19:43:20.099053 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093199    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 05 19:43:20.099053 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093202    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 05 19:43:20.099053 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093205    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 05 19:43:20.099053 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093207    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 05 19:43:20.099053 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093210    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 05 19:43:20.099053 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093212    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 05 19:43:20.099053 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093216    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 05 19:43:20.099053 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093222    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 05 19:43:20.099053 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093226    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 05 19:43:20.099053 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093229    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 05 19:43:20.099053 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093231    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 05 19:43:20.099053 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093234    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 05 19:43:20.099053 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093236    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 05 19:43:20.099053 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093239    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 05 19:43:20.099053 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093241    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 05 19:43:20.099053 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093244    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 05 19:43:20.099053 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093246    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 05 19:43:20.099053 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093252    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 05 19:43:20.099053 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093255    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 05 19:43:20.099601 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093259    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 05 19:43:20.099601 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093262    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 05 19:43:20.099601 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093265    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 05 19:43:20.099601 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093269    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 05 19:43:20.099601 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093271    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 05 19:43:20.099601 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093274    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 05 19:43:20.099601 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093278    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 05 19:43:20.099601 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093281    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 05 19:43:20.099601 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:20.093284    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 05 19:43:20.099601 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.093288    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 05 19:43:20.099601 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.093409    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 05 19:43:20.099601 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.097290    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 05 19:43:20.099601 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.098263    2583 server.go:1019] "Starting client certificate rotation"
Sep 05 19:43:20.099601 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.098365    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 05 19:43:20.099958 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.098397    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 05 19:43:20.100774 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.100761    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 05 19:43:20.101936 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.101919    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 05 19:43:20.114485 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.114468    2583 log.go:25] "Validated CRI v1 runtime API"
Sep 05 19:43:20.118014 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.118001    2583 log.go:25] "Validated CRI v1 image API"
Sep 05 19:43:20.120194 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.120175    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 05 19:43:20.122475 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.122455    2583 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b689d611-f6b7-4603-8c95-710d18bee9b7:/dev/nvme0n1p4 ee730c12-5ce7-4391-a04e-a96e7afde424:/dev/nvme0n1p3]
Sep 05 19:43:20.122566 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.122473    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 05 19:43:20.127312 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.127212    2583 manager.go:216] Machine: {Timestamp:2026-09-05 19:43:20.125981849 +0000 UTC m=+0.330205393 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100144 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec267e688fb2ca1435a4ab66f477b3aa SystemUUID:ec267e68-8fb2-ca14-35a4-ab66f477b3aa BootID:6e2a1de8-e7d2-4a61-ae90-490bf2642070 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b6:25:83:0d:01 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b6:25:83:0d:01 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:c0:1f:23:37:6e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 05 19:43:20.127312 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.127306    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 05 19:43:20.128019 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.127566    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 05 19:43:20.128672 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.128643    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 05 19:43:20.131465 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.131431    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 05 19:43:20.131677 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.131470    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-208.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 05 19:43:20.132770 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.132759    2583 topology_manager.go:138] "Creating topology manager with none policy"
Sep 05 19:43:20.132805 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.132773    2583 container_manager_linux.go:309] "Creating device plugin manager"
Sep 05 19:43:20.132805 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.132781    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 05 19:43:20.133353 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.133343    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 05 19:43:20.134589 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.134578    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 05 19:43:20.138056 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.138045    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 05 19:43:20.138180 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.138169    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 05 19:43:20.138210 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.138202    2583 kubelet.go:486] "Attempting to sync node with API server"
Sep 05 19:43:20.138238 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.138213    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 05 19:43:20.138238 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.138227    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 05 19:43:20.138238 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.138237    2583 kubelet.go:398] "Adding apiserver pod source"
Sep 05 19:43:20.138352 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.138245    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 05 19:43:20.138624 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.138612    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 05 19:43:20.138677 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.138629    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 05 19:43:20.140714 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.140695    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 05 19:43:20.142738 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.142722    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 05 19:43:20.142819 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.142770    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 05 19:43:20.142976 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.142964    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 05 19:43:20.143018 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.142985    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 05 19:43:20.143018 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.142995    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 05 19:43:20.143018 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.143003    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 05 19:43:20.143018 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.143013    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 05 19:43:20.143120 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.143022    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 05 19:43:20.143337 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.143327    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 05 19:43:20.143366 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.143338    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 05 19:43:20.143366 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.143346    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 05 19:43:20.143366 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.143352    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 05 19:43:20.143366 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.143361    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 05 19:43:20.143465 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.143368    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 05 19:43:20.143931 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.143921    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 05 19:43:20.143962 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.143933    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 05 19:43:20.147835 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:20.147810    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-208.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 05 19:43:20.147835 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.147820    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-208.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 05 19:43:20.147952 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:20.147868    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 05 19:43:20.148004 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.147967    2583 server.go:1268] "Started kubelet"
Sep 05 19:43:20.148135 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.148044    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 05 19:43:20.148189 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.148153    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 05 19:43:20.148189 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.148062    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 05 19:43:20.148869 ip-10-0-142-208 systemd[1]: Started Kubernetes Kubelet.
Sep 05 19:43:20.150068 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.150042    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 05 19:43:20.152498 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.151802    2583 server.go:310] "Adding debug handlers to kubelet server"
Sep 05 19:43:20.153711 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:20.153691    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 05 19:43:20.155078 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.155065    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 05 19:43:20.155531 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.155502    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 05 19:43:20.155731 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:20.155707    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-208.ec2.internal\" not found"
Sep 05 19:43:20.156141 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.156118    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 05 19:43:20.156221 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.156148    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 05 19:43:20.156221 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.156118    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 05 19:43:20.156303 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.156244    2583 reconstruct.go:97] "Volume reconstruction finished"
Sep 05 19:43:20.156303 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.156252    2583 reconciler.go:29] "Reconciler: start to sync state"
Sep 05 19:43:20.156381 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.156365    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 05 19:43:20.156381 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.156377    2583 factory.go:55] Registering systemd factory
Sep 05 19:43:20.156467 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.156386    2583 factory.go:223] Registration of the systemd container factory successfully
Sep 05 19:43:20.156611 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.156598    2583 factory.go:153] Registering CRI-O factory
Sep 05 19:43:20.156611 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.156611    2583 factory.go:223] Registration of the crio container factory successfully
Sep 05 19:43:20.156703 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.156624    2583 factory.go:103] Registering Raw factory
Sep 05 19:43:20.156703 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.156635    2583 manager.go:1196] Started watching for new ooms in manager
Sep 05 19:43:20.156947 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.156934    2583 manager.go:349] Starting recovery of all containers
Sep 05 19:43:20.156947 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:20.156035    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-208.ec2.internal.18d2843d23cbdbd9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-208.ec2.internal,UID:ip-10-0-142-208.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-208.ec2.internal,},FirstTimestamp:2026-09-05 19:43:20.147942361 +0000 UTC m=+0.352165906,LastTimestamp:2026-09-05 19:43:20.147942361 +0000 UTC m=+0.352165906,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-208.ec2.internal,}"
Sep 05 19:43:20.164867 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.164647    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fhqfn"
Sep 05 19:43:20.165879 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:20.165850    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-208.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 05 19:43:20.165979 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:20.165901    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 05 19:43:20.170897 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.170879    2583 manager.go:354] Recovery completed
Sep 05 19:43:20.172223 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:20.172130    2583 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 05 19:43:20.175691 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.175671    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fhqfn"
Sep 05 19:43:20.176402 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.176391    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 19:43:20.178265 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.178245    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-208.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 19:43:20.178340 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.178278    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-208.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 19:43:20.178340 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.178292    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-208.ec2.internal" event="NodeHasSufficientPID"
Sep 05 19:43:20.178898 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.178883    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 05 19:43:20.178898 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.178898    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 05 19:43:20.179050 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.178902    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 19:43:20.179050 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.178916    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 05 19:43:20.179646 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.179630    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-208.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 19:43:20.179702 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.179659    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-208.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 19:43:20.179702 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.179668    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-208.ec2.internal" event="NodeHasSufficientPID"
Sep 05 19:43:20.180919 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.180905    2583 policy_none.go:49] "None policy: Start"
Sep 05 19:43:20.180919 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.180922    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 05 19:43:20.181041 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.180932    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 05 19:43:20.182873 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.182861    2583 policy_none.go:47] "Start"
Sep 05 19:43:20.328698 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.216173    2583 manager.go:337] "Starting Device Plugin manager"
Sep 05 19:43:20.328698 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:20.216212    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 05 19:43:20.328698 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.216225    2583 server.go:87] "Starting device plugin registration server"
Sep 05 19:43:20.328698 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.216561    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 05 19:43:20.328698 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.216571    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 05 19:43:20.328698 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.216657    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 05 19:43:20.328698 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.216737    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 05 19:43:20.328698 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.216745    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 05 19:43:20.328698 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:20.217370    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 05 19:43:20.328698 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:20.217432    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-208.ec2.internal\" not found"
Sep 05 19:43:20.328698 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.252167    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 05 19:43:20.328698 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.253354    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 05 19:43:20.328698 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.253395    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 05 19:43:20.328698 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.253418    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 05 19:43:20.328698 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:20.253460    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 05 19:43:20.328698 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.256456    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 19:43:20.328698 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.316940    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 19:43:20.328698 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.318252    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-208.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 19:43:20.328698 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.318299    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-208.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 19:43:20.328698 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.318325    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-208.ec2.internal" event="NodeHasSufficientPID"
Sep 05 19:43:20.328698 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.318372    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-208.ec2.internal"
Sep 05 19:43:20.328698 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.325747    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-208.ec2.internal"
Sep 05 19:43:20.328698 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:20.325765    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-208.ec2.internal\": node \"ip-10-0-142-208.ec2.internal\" not found"
Sep 05 19:43:20.345732 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:20.345708    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-208.ec2.internal\" not found"
Sep 05 19:43:20.354419 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.354401    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-208.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-208.ec2.internal"]
Sep 05 19:43:20.354495 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.354465    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 19:43:20.355278 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.355262    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-208.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 19:43:20.355361 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.355293    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-208.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 19:43:20.355361 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.355306    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-208.ec2.internal" event="NodeHasSufficientPID"
Sep 05 19:43:20.357615 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.357603    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 19:43:20.357708 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.357695    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-208.ec2.internal"
Sep 05 19:43:20.357750 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.357724    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 19:43:20.358292 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.358273    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-208.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 19:43:20.358292 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.358291    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-208.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 19:43:20.358407 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.358303    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-208.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 19:43:20.358407 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.358312    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-208.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 19:43:20.358407 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.358319    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-208.ec2.internal" event="NodeHasSufficientPID"
Sep 05 19:43:20.358407 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.358322    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-208.ec2.internal" event="NodeHasSufficientPID"
Sep 05 19:43:20.360468 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.360453    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-208.ec2.internal"
Sep 05 19:43:20.360599 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.360476    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 05 19:43:20.361114 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.361101    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-208.ec2.internal" event="NodeHasSufficientMemory"
Sep 05 19:43:20.361175 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.361125    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-208.ec2.internal" event="NodeHasNoDiskPressure"
Sep 05 19:43:20.361175 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.361135    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-208.ec2.internal" event="NodeHasSufficientPID"
Sep 05 19:43:20.383142 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:20.383120    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-208.ec2.internal\" not found" node="ip-10-0-142-208.ec2.internal"
Sep 05 19:43:20.386497 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:20.386483    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-208.ec2.internal\" not found" node="ip-10-0-142-208.ec2.internal"
Sep 05 19:43:20.445807 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:20.445782    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-208.ec2.internal\" not found"
Sep 05 19:43:20.457085 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.457064    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/78a3deb210d787366a6f198baa73b2ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-208.ec2.internal\" (UID: \"78a3deb210d787366a6f198baa73b2ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-208.ec2.internal"
Sep 05 19:43:20.457161 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.457089    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78a3deb210d787366a6f198baa73b2ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-208.ec2.internal\" (UID: \"78a3deb210d787366a6f198baa73b2ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-208.ec2.internal"
Sep 05 19:43:20.545881 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:20.545851    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-208.ec2.internal\" not found"
Sep 05 19:43:20.558190 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.558154    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/78a3deb210d787366a6f198baa73b2ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-208.ec2.internal\" (UID: \"78a3deb210d787366a6f198baa73b2ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-208.ec2.internal"
Sep 05 19:43:20.558331 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.558198    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78a3deb210d787366a6f198baa73b2ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-208.ec2.internal\" (UID: \"78a3deb210d787366a6f198baa73b2ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-208.ec2.internal"
Sep 05 19:43:20.558331 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.558230    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef30841ed7c762f04a60906c21dc14a2-config\") pod \"kube-apiserver-proxy-ip-10-0-142-208.ec2.internal\" (UID: \"ef30841ed7c762f04a60906c21dc14a2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-208.ec2.internal"
Sep 05 19:43:20.558331 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.558267    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78a3deb210d787366a6f198baa73b2ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-208.ec2.internal\" (UID: \"78a3deb210d787366a6f198baa73b2ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-208.ec2.internal"
Sep 05 19:43:20.558331 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.558265    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/78a3deb210d787366a6f198baa73b2ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-208.ec2.internal\" (UID: \"78a3deb210d787366a6f198baa73b2ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-208.ec2.internal"
Sep 05 19:43:20.646694 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:20.646635    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-208.ec2.internal\" not found"
Sep 05 19:43:20.659052 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.659030    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef30841ed7c762f04a60906c21dc14a2-config\") pod \"kube-apiserver-proxy-ip-10-0-142-208.ec2.internal\" (UID: \"ef30841ed7c762f04a60906c21dc14a2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-208.ec2.internal"
Sep 05 19:43:20.659145 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.659073    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef30841ed7c762f04a60906c21dc14a2-config\") pod \"kube-apiserver-proxy-ip-10-0-142-208.ec2.internal\" (UID: \"ef30841ed7c762f04a60906c21dc14a2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-208.ec2.internal"
Sep 05 19:43:20.686187 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.686157    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-208.ec2.internal"
Sep 05 19:43:20.689337 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.689315    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-208.ec2.internal"
Sep 05 19:43:20.746792 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:20.746758    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-208.ec2.internal\" not found"
Sep 05 19:43:20.847316 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:20.847289    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-208.ec2.internal\" not found"
Sep 05 19:43:20.947842 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:20.947774    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-208.ec2.internal\" not found"
Sep 05 19:43:20.975083 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:20.975064    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 19:43:22.230504 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.230479    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-208.ec2.internal"
Sep 05 19:43:22.250953 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.250930    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 05 19:43:22.254923 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.254899    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-208.ec2.internal"
Sep 05 19:43:22.265910 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.265892    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 05 19:43:22.273015 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.272997    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 05 19:43:22.273151 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.273131    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 05 19:43:22.273206 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.273170    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 05 19:43:22.313325 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.313303    2583 apiserver.go:52] "Watching apiserver"
Sep 05 19:43:22.325432 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.325413    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 05 19:43:22.325790 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.325772    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-208.ec2.internal","openshift-dns/node-resolver-5h45d","openshift-image-registry/node-ca-z6kd9","openshift-multus/network-metrics-daemon-rwwbl","openshift-network-operator/iptables-alerter-mhzdb","openshift-ovn-kubernetes/ovnkube-node-mf9tj","kube-system/global-pull-secret-syncer-ccth2","kube-system/konnectivity-agent-sdb8j","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng","openshift-cluster-node-tuning-operator/tuned-mpjps","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-208.ec2.internal","openshift-multus/multus-additional-cni-plugins-wzs5q","openshift-multus/multus-grttc","openshift-network-diagnostics/network-check-target-zxsbh"]
Sep 05 19:43:22.327922 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.327904    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:22.328025 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:22.327978    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccth2" podUID="83377d0d-0308-4613-bd53-04d2cb4adc3b"
Sep 05 19:43:22.329807 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.329793    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 05 19:43:22.329932 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.329914    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.331918 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.331901    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z6kd9"
Sep 05 19:43:22.333219 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.333199    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 05 19:43:22.334366 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.333958    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:22.334366 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.333990    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8l74k\""
Sep 05 19:43:22.334366 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.334001    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 05 19:43:22.334366 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:22.334031    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rwwbl" podUID="b9720241-ac7b-4d79-9644-233caff66a8c"
Sep 05 19:43:22.334366 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.334057    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 05 19:43:22.334366 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.334176    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 05 19:43:22.334366 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.334182    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 05 19:43:22.334366 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.334232    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 05 19:43:22.334366 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.334268    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 05 19:43:22.334895 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.334732    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jzh4k\""
Sep 05 19:43:22.336475 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.336455    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mhzdb"
Sep 05 19:43:22.337061 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.337008    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-host-run-netns\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.337061 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.337037    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-hostroot\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.337236 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.337075    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxm26\" (UniqueName: \"kubernetes.io/projected/02b1f61a-c064-4c08-9bd4-cbe7fe25c934-kube-api-access-pxm26\") pod \"node-ca-z6kd9\" (UID: \"02b1f61a-c064-4c08-9bd4-cbe7fe25c934\") " pod="openshift-image-registry/node-ca-z6kd9"
Sep 05 19:43:22.337236 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.337104    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-multus-socket-dir-parent\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.337236 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.337149    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-host-var-lib-cni-multus\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.337236 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.337172    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-multus-conf-dir\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.337236 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.337193    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-multus-daemon-config\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.337236 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.337208    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-host-run-multus-certs\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.337236 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.337225    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-etc-kubernetes\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.337533 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.337259    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/83377d0d-0308-4613-bd53-04d2cb4adc3b-dbus\") pod \"global-pull-secret-syncer-ccth2\" (UID: \"83377d0d-0308-4613-bd53-04d2cb4adc3b\") " pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:22.337533 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.337275    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83377d0d-0308-4613-bd53-04d2cb4adc3b-original-pull-secret\") pod \"global-pull-secret-syncer-ccth2\" (UID: \"83377d0d-0308-4613-bd53-04d2cb4adc3b\") " pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:22.337533 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.337291    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-cni-binary-copy\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.337533 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.337333    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-host-var-lib-kubelet\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.337533 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.337347    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02b1f61a-c064-4c08-9bd4-cbe7fe25c934-host\") pod \"node-ca-z6kd9\" (UID: \"02b1f61a-c064-4c08-9bd4-cbe7fe25c934\") " pod="openshift-image-registry/node-ca-z6kd9"
Sep 05 19:43:22.337533 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.337391    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02b1f61a-c064-4c08-9bd4-cbe7fe25c934-serviceca\") pod \"node-ca-z6kd9\" (UID: \"02b1f61a-c064-4c08-9bd4-cbe7fe25c934\") " pod="openshift-image-registry/node-ca-z6kd9"
Sep 05 19:43:22.337533 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.337429    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/83377d0d-0308-4613-bd53-04d2cb4adc3b-kubelet-config\") pod \"global-pull-secret-syncer-ccth2\" (UID: \"83377d0d-0308-4613-bd53-04d2cb4adc3b\") " pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:22.337533 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.337457    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-system-cni-dir\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.337533 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.337482    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-cnibin\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.337533 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.337506    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-host-run-k8s-cni-cncf-io\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.337533 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.337531    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-host-var-lib-cni-bin\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.338006 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.337571    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nd9j\" (UniqueName: \"kubernetes.io/projected/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-kube-api-access-6nd9j\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.338006 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.337597    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-multus-cni-dir\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.338712 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.338612    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 05 19:43:22.338858 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.338842    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 05 19:43:22.338924 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.338908    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rg6jh\""
Sep 05 19:43:22.338970 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.338940    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 05 19:43:22.340068 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.340049    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.340873 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.340855    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 05 19:43:22.341705 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.341690    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g4q8d\""
Sep 05 19:43:22.341909 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.341887    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng"
Sep 05 19:43:22.341991 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.341912    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 05 19:43:22.342057 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.342001    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 05 19:43:22.342502 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.342479    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 05 19:43:22.343190 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.343168    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 05 19:43:22.343458 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.343440    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 05 19:43:22.343654 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.343633    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 05 19:43:22.343998 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.343976    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 05 19:43:22.345586 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.345555    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 05 19:43:22.345994 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.345977    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 05 19:43:22.346080 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.346068    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m5n9z\""
Sep 05 19:43:22.346157 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.346075    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:22.346230 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:22.346175    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxsbh" podUID="63646bb5-7fb0-417d-96a8-24d5c8591bf1"
Sep 05 19:43:22.348431 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.348413    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5h45d"
Sep 05 19:43:22.350477 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.350459    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 05 19:43:22.350569 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.350509    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ms42p\""
Sep 05 19:43:22.350569 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.350527    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 05 19:43:22.350780 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.350766    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.352600 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.352583    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 05 19:43:22.352816 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.352800    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nxtk4\""
Sep 05 19:43:22.352933 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.352917    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 05 19:43:22.353095 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.353033    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-04 19:38:21 +0000 UTC" deadline="2028-03-25 19:08:03.273292019 +0000 UTC"
Sep 05 19:43:22.353216 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.353094    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13607h24m40.92020078s"
Sep 05 19:43:22.353216 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.353167    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sdb8j"
Sep 05 19:43:22.355159 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.355143    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 05 19:43:22.355159 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.355155    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c2ggz\""
Sep 05 19:43:22.355251 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.355188    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 05 19:43:22.355515 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.355501    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wzs5q"
Sep 05 19:43:22.358332 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.358313    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 05 19:43:22.358584 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.358572    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 05 19:43:22.358980 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.358968    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5vbt8\""
Sep 05 19:43:22.379904 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.379860    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hzgzx"
Sep 05 19:43:22.390156 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.390136    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hzgzx"
Sep 05 19:43:22.409531 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.409515    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 19:43:22.432389 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.432371    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 05 19:43:22.438193 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438174    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9398b13d-ef3a-4d9e-96f4-725c11892e70-device-dir\") pod \"aws-ebs-csi-driver-node-xnjng\" (UID: \"9398b13d-ef3a-4d9e-96f4-725c11892e70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng"
Sep 05 19:43:22.438287 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438207    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-host-run-netns\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.438287 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438230    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-lib-modules\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.438287 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438251    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-etc-openvswitch\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.438287 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438274    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckqjv\" (UniqueName: \"kubernetes.io/projected/63646bb5-7fb0-417d-96a8-24d5c8591bf1-kube-api-access-ckqjv\") pod \"network-check-target-zxsbh\" (UID: \"63646bb5-7fb0-417d-96a8-24d5c8591bf1\") " pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:22.438465 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438299    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-multus-socket-dir-parent\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.438465 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438298    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-host-run-netns\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.438465 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438353    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0216ba9d-7122-47a3-86eb-8d14ddb478b0-tmp-dir\") pod \"node-resolver-5h45d\" (UID: \"0216ba9d-7122-47a3-86eb-8d14ddb478b0\") " pod="openshift-dns/node-resolver-5h45d"
Sep 05 19:43:22.438465 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438377    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-multus-socket-dir-parent\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.438465 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438383    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-etc-kubernetes\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.438465 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438442    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f4df1d96-03b4-4947-aa18-86dd66548441-agent-certs\") pod \"konnectivity-agent-sdb8j\" (UID: \"f4df1d96-03b4-4947-aa18-86dd66548441\") " pod="kube-system/konnectivity-agent-sdb8j"
Sep 05 19:43:22.438465 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438465    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d0749ab-114b-4d89-920d-60979f8b485e-host-slash\") pod \"iptables-alerter-mhzdb\" (UID: \"4d0749ab-114b-4d89-920d-60979f8b485e\") " pod="openshift-network-operator/iptables-alerter-mhzdb"
Sep 05 19:43:22.438702 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438486    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-host-cni-bin\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.438702 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438508    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbn8f\" (UniqueName: \"kubernetes.io/projected/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-kube-api-access-tbn8f\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.438702 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438538    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83377d0d-0308-4613-bd53-04d2cb4adc3b-original-pull-secret\") pod \"global-pull-secret-syncer-ccth2\" (UID: \"83377d0d-0308-4613-bd53-04d2cb4adc3b\") " pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:22.438702 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438583    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-cni-binary-copy\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.438702 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:22.438688    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:22.438904 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438705    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02b1f61a-c064-4c08-9bd4-cbe7fe25c934-serviceca\") pod \"node-ca-z6kd9\" (UID: \"02b1f61a-c064-4c08-9bd4-cbe7fe25c934\") " pod="openshift-image-registry/node-ca-z6kd9"
Sep 05 19:43:22.438904 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:22.438778    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83377d0d-0308-4613-bd53-04d2cb4adc3b-original-pull-secret podName:83377d0d-0308-4613-bd53-04d2cb4adc3b nodeName:}" failed. No retries permitted until 2026-09-05 19:43:22.938742961 +0000 UTC m=+1.968374861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83377d0d-0308-4613-bd53-04d2cb4adc3b-original-pull-secret") pod "global-pull-secret-syncer-ccth2" (UID: "83377d0d-0308-4613-bd53-04d2cb4adc3b") : object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:22.438904 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438796    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-run-ovn\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.438904 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438828    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-log-socket\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.438904 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438864    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-ovn-node-metrics-cert\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.438904 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438895    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/83377d0d-0308-4613-bd53-04d2cb4adc3b-kubelet-config\") pod \"global-pull-secret-syncer-ccth2\" (UID: \"83377d0d-0308-4613-bd53-04d2cb4adc3b\") " pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:22.439230 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438921    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-system-cni-dir\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.439230 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438946    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nd9j\" (UniqueName: \"kubernetes.io/projected/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-kube-api-access-6nd9j\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.439230 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438968    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-etc-sysctl-d\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.439230 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.438992    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/334781cf-bbb8-46db-a50c-4d13f6e9f204-cnibin\") pod \"multus-additional-cni-plugins-wzs5q\" (UID: \"334781cf-bbb8-46db-a50c-4d13f6e9f204\") " pod="openshift-multus/multus-additional-cni-plugins-wzs5q"
Sep 05 19:43:22.439230 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.439018    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/334781cf-bbb8-46db-a50c-4d13f6e9f204-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wzs5q\" (UID: \"334781cf-bbb8-46db-a50c-4d13f6e9f204\") " pod="openshift-multus/multus-additional-cni-plugins-wzs5q"
Sep 05 19:43:22.439230 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.439045    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-run-systemd\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.439230 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.439063    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-system-cni-dir\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.439230 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.439074    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-node-log\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.439230 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.439103    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/27ce30e4-5b20-4020-a55e-c9909e94e228-etc-tuned\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.439230 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.439143    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgvcm\" (UniqueName: \"kubernetes.io/projected/4d0749ab-114b-4d89-920d-60979f8b485e-kube-api-access-bgvcm\") pod \"iptables-alerter-mhzdb\" (UID: \"4d0749ab-114b-4d89-920d-60979f8b485e\") " pod="openshift-network-operator/iptables-alerter-mhzdb"
Sep 05 19:43:22.439230 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.439167    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-ovnkube-script-lib\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.439230 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.439196    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4d0749ab-114b-4d89-920d-60979f8b485e-iptables-alerter-script\") pod \"iptables-alerter-mhzdb\" (UID: \"4d0749ab-114b-4d89-920d-60979f8b485e\") " pod="openshift-network-operator/iptables-alerter-mhzdb"
Sep 05 19:43:22.439230 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.439226    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-multus-daemon-config\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.439230 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.439255    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-cni-binary-copy\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.439230 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.439276    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0216ba9d-7122-47a3-86eb-8d14ddb478b0-hosts-file\") pod \"node-resolver-5h45d\" (UID: \"0216ba9d-7122-47a3-86eb-8d14ddb478b0\") " pod="openshift-dns/node-resolver-5h45d"
Sep 05 19:43:22.439927 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.439336    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-etc-sysconfig\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.439927 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.439369    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-var-lib-openvswitch\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.439927 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.439397    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-host-run-ovn-kubernetes\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.439927 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.439443    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9398b13d-ef3a-4d9e-96f4-725c11892e70-registration-dir\") pod \"aws-ebs-csi-driver-node-xnjng\" (UID: \"9398b13d-ef3a-4d9e-96f4-725c11892e70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng"
Sep 05 19:43:22.439927 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.439470    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9398b13d-ef3a-4d9e-96f4-725c11892e70-etc-selinux\") pod \"aws-ebs-csi-driver-node-xnjng\" (UID: \"9398b13d-ef3a-4d9e-96f4-725c11892e70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng"
Sep 05 19:43:22.439927 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.439477    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/83377d0d-0308-4613-bd53-04d2cb4adc3b-kubelet-config\") pod \"global-pull-secret-syncer-ccth2\" (UID: \"83377d0d-0308-4613-bd53-04d2cb4adc3b\") " pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:22.439927 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.439829    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02b1f61a-c064-4c08-9bd4-cbe7fe25c934-host\") pod \"node-ca-z6kd9\" (UID: \"02b1f61a-c064-4c08-9bd4-cbe7fe25c934\") " pod="openshift-image-registry/node-ca-z6kd9"
Sep 05 19:43:22.440394 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.440179    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgfpq\" (UniqueName: \"kubernetes.io/projected/0216ba9d-7122-47a3-86eb-8d14ddb478b0-kube-api-access-bgfpq\") pod \"node-resolver-5h45d\" (UID: \"0216ba9d-7122-47a3-86eb-8d14ddb478b0\") " pod="openshift-dns/node-resolver-5h45d"
Sep 05 19:43:22.440394 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.440209    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-run\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.440394 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.440206    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02b1f61a-c064-4c08-9bd4-cbe7fe25c934-serviceca\") pod \"node-ca-z6kd9\" (UID: \"02b1f61a-c064-4c08-9bd4-cbe7fe25c934\") " pod="openshift-image-registry/node-ca-z6kd9"
Sep 05 19:43:22.440394 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.440272    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-host-kubelet\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.440394 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.440315    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.440394 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.440318    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02b1f61a-c064-4c08-9bd4-cbe7fe25c934-host\") pod \"node-ca-z6kd9\" (UID: \"02b1f61a-c064-4c08-9bd4-cbe7fe25c934\") " pod="openshift-image-registry/node-ca-z6kd9"
Sep 05 19:43:22.440686 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.440381    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9398b13d-ef3a-4d9e-96f4-725c11892e70-socket-dir\") pod \"aws-ebs-csi-driver-node-xnjng\" (UID: \"9398b13d-ef3a-4d9e-96f4-725c11892e70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng"
Sep 05 19:43:22.440686 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.440582    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-cnibin\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.440686 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.440603    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-etc-sysctl-conf\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.440686 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.440621    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-var-lib-kubelet\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.440686 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.440632    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-multus-daemon-config\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.440686 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.440639    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/334781cf-bbb8-46db-a50c-4d13f6e9f204-cni-binary-copy\") pod \"multus-additional-cni-plugins-wzs5q\" (UID: \"334781cf-bbb8-46db-a50c-4d13f6e9f204\") " pod="openshift-multus/multus-additional-cni-plugins-wzs5q"
Sep 05 19:43:22.440686 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.440679    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-cnibin\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.441033 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.440709    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-multus-cni-dir\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.441033 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.440739    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-hostroot\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.441033 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.440784    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gzqd\" (UniqueName: \"kubernetes.io/projected/27ce30e4-5b20-4020-a55e-c9909e94e228-kube-api-access-2gzqd\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.441033 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.440817    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-env-overrides\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.441033 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.440819    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-multus-cni-dir\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.441033 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.440850    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9398b13d-ef3a-4d9e-96f4-725c11892e70-sys-fs\") pod \"aws-ebs-csi-driver-node-xnjng\" (UID: \"9398b13d-ef3a-4d9e-96f4-725c11892e70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng"
Sep 05 19:43:22.441033 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.440848    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-hostroot\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.441033 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.440894    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc4hg\" (UniqueName: \"kubernetes.io/projected/b9720241-ac7b-4d79-9644-233caff66a8c-kube-api-access-fc4hg\") pod \"network-metrics-daemon-rwwbl\" (UID: \"b9720241-ac7b-4d79-9644-233caff66a8c\") " pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:22.441033 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.440930    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f4df1d96-03b4-4947-aa18-86dd66548441-konnectivity-ca\") pod \"konnectivity-agent-sdb8j\" (UID: \"f4df1d96-03b4-4947-aa18-86dd66548441\") " pod="kube-system/konnectivity-agent-sdb8j"
Sep 05 19:43:22.441033 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.440959    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/334781cf-bbb8-46db-a50c-4d13f6e9f204-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wzs5q\" (UID: \"334781cf-bbb8-46db-a50c-4d13f6e9f204\") " pod="openshift-multus/multus-additional-cni-plugins-wzs5q"
Sep 05 19:43:22.441033 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.440992    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-host-run-netns\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.441033 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441023    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-run-openvswitch\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.441594 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441074    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/83377d0d-0308-4613-bd53-04d2cb4adc3b-dbus\") pod \"global-pull-secret-syncer-ccth2\" (UID: \"83377d0d-0308-4613-bd53-04d2cb4adc3b\") " pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:22.441594 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441133    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-etc-modprobe-d\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.441594 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441164    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-etc-systemd\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.441594 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441205    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/334781cf-bbb8-46db-a50c-4d13f6e9f204-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wzs5q\" (UID: \"334781cf-bbb8-46db-a50c-4d13f6e9f204\") " pod="openshift-multus/multus-additional-cni-plugins-wzs5q"
Sep 05 19:43:22.441594 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441235    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-host-cni-netd\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.441594 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441266    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59264\" (UniqueName: \"kubernetes.io/projected/9398b13d-ef3a-4d9e-96f4-725c11892e70-kube-api-access-59264\") pod \"aws-ebs-csi-driver-node-xnjng\" (UID: \"9398b13d-ef3a-4d9e-96f4-725c11892e70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng"
Sep 05 19:43:22.441594 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441308    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-host-run-k8s-cni-cncf-io\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.441594 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441336    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-host-var-lib-cni-bin\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.441594 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441366    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs\") pod \"network-metrics-daemon-rwwbl\" (UID: \"b9720241-ac7b-4d79-9644-233caff66a8c\") " pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:22.441594 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441394    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27ce30e4-5b20-4020-a55e-c9909e94e228-tmp\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.441594 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441420    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-ovnkube-config\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.441594 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441450    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9398b13d-ef3a-4d9e-96f4-725c11892e70-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xnjng\" (UID: \"9398b13d-ef3a-4d9e-96f4-725c11892e70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng"
Sep 05 19:43:22.441594 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441514    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxm26\" (UniqueName: \"kubernetes.io/projected/02b1f61a-c064-4c08-9bd4-cbe7fe25c934-kube-api-access-pxm26\") pod \"node-ca-z6kd9\" (UID: \"02b1f61a-c064-4c08-9bd4-cbe7fe25c934\") " pod="openshift-image-registry/node-ca-z6kd9"
Sep 05 19:43:22.441594 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441541    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq95c\" (UniqueName: \"kubernetes.io/projected/334781cf-bbb8-46db-a50c-4d13f6e9f204-kube-api-access-gq95c\") pod \"multus-additional-cni-plugins-wzs5q\" (UID: \"334781cf-bbb8-46db-a50c-4d13f6e9f204\") " pod="openshift-multus/multus-additional-cni-plugins-wzs5q"
Sep 05 19:43:22.441594 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441571    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-host-slash\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.442268 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441609    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-host-var-lib-cni-multus\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.442268 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441640    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-multus-conf-dir\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.442268 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441675    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-host-run-multus-certs\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.442268 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441724    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-etc-kubernetes\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.442268 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441753    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-host\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.442268 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441783    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-systemd-units\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.442268 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441817    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-host-var-lib-kubelet\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.442268 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441952    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-sys\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.442268 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.441979    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/334781cf-bbb8-46db-a50c-4d13f6e9f204-system-cni-dir\") pod \"multus-additional-cni-plugins-wzs5q\" (UID: \"334781cf-bbb8-46db-a50c-4d13f6e9f204\") " pod="openshift-multus/multus-additional-cni-plugins-wzs5q"
Sep 05 19:43:22.442268 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.442104    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-host-var-lib-kubelet\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.442268 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.442191    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/83377d0d-0308-4613-bd53-04d2cb4adc3b-dbus\") pod \"global-pull-secret-syncer-ccth2\" (UID: \"83377d0d-0308-4613-bd53-04d2cb4adc3b\") " pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:22.442268 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.442244    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-host-run-k8s-cni-cncf-io\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.442777 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.442285    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-host-var-lib-cni-bin\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.442777 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.442571    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-host-var-lib-cni-multus\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.442777 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.442611    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-multus-conf-dir\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.442777 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.442662    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-host-run-multus-certs\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.442777 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.442707    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-etc-kubernetes\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.454187 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.454166    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 05 19:43:22.457879 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.457761    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nd9j\" (UniqueName: \"kubernetes.io/projected/2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3-kube-api-access-6nd9j\") pod \"multus-grttc\" (UID: \"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3\") " pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.457945 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.457837    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxm26\" (UniqueName: \"kubernetes.io/projected/02b1f61a-c064-4c08-9bd4-cbe7fe25c934-kube-api-access-pxm26\") pod \"node-ca-z6kd9\" (UID: \"02b1f61a-c064-4c08-9bd4-cbe7fe25c934\") " pod="openshift-image-registry/node-ca-z6kd9"
Sep 05 19:43:22.462901 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:22.462326    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a3deb210d787366a6f198baa73b2ab.slice/crio-0fec068c65d5b98dbcbf1a3a3efb584bfe25b7904467848069748fac2f8f19f9 WatchSource:0}: Error finding container 0fec068c65d5b98dbcbf1a3a3efb584bfe25b7904467848069748fac2f8f19f9: Status 404 returned error can't find the container with id 0fec068c65d5b98dbcbf1a3a3efb584bfe25b7904467848069748fac2f8f19f9
Sep 05 19:43:22.462996 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:22.462848    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef30841ed7c762f04a60906c21dc14a2.slice/crio-979676b4adff5ebd2ad94f6f7d91e486cfc3c0d15abc8c02d58f3a4e778dd50d WatchSource:0}: Error finding container 979676b4adff5ebd2ad94f6f7d91e486cfc3c0d15abc8c02d58f3a4e778dd50d: Status 404 returned error can't find the container with id 979676b4adff5ebd2ad94f6f7d91e486cfc3c0d15abc8c02d58f3a4e778dd50d
Sep 05 19:43:22.468022 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.468001    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 19:43:22.543245 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543220    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-run-ovn\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.543245 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543246    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-log-socket\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.543443 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543263    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-ovn-node-metrics-cert\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.543443 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543280    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-etc-sysctl-d\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.543443 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543294    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/334781cf-bbb8-46db-a50c-4d13f6e9f204-cnibin\") pod \"multus-additional-cni-plugins-wzs5q\" (UID: \"334781cf-bbb8-46db-a50c-4d13f6e9f204\") " pod="openshift-multus/multus-additional-cni-plugins-wzs5q"
Sep 05 19:43:22.543443 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543317    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/334781cf-bbb8-46db-a50c-4d13f6e9f204-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wzs5q\" (UID: \"334781cf-bbb8-46db-a50c-4d13f6e9f204\") " pod="openshift-multus/multus-additional-cni-plugins-wzs5q"
Sep 05 19:43:22.543443 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543333    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-log-socket\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.543443 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543333    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-run-ovn\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.543443 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543340    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-run-systemd\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.543443 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543380    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-run-systemd\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.543443 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543390    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-node-log\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.543443 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543394    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/334781cf-bbb8-46db-a50c-4d13f6e9f204-cnibin\") pod \"multus-additional-cni-plugins-wzs5q\" (UID: \"334781cf-bbb8-46db-a50c-4d13f6e9f204\") " pod="openshift-multus/multus-additional-cni-plugins-wzs5q"
Sep 05 19:43:22.543443 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543414    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/27ce30e4-5b20-4020-a55e-c9909e94e228-etc-tuned\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.543443 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543437    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-node-log\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.543443 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543442    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgvcm\" (UniqueName: \"kubernetes.io/projected/4d0749ab-114b-4d89-920d-60979f8b485e-kube-api-access-bgvcm\") pod \"iptables-alerter-mhzdb\" (UID: \"4d0749ab-114b-4d89-920d-60979f8b485e\") " pod="openshift-network-operator/iptables-alerter-mhzdb"
Sep 05 19:43:22.544023 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543472    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-ovnkube-script-lib\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.544023 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543497    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-etc-sysctl-d\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.544023 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543538    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4d0749ab-114b-4d89-920d-60979f8b485e-iptables-alerter-script\") pod \"iptables-alerter-mhzdb\" (UID: \"4d0749ab-114b-4d89-920d-60979f8b485e\") " pod="openshift-network-operator/iptables-alerter-mhzdb"
Sep 05 19:43:22.544023 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543625    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0216ba9d-7122-47a3-86eb-8d14ddb478b0-hosts-file\") pod \"node-resolver-5h45d\" (UID: \"0216ba9d-7122-47a3-86eb-8d14ddb478b0\") " pod="openshift-dns/node-resolver-5h45d"
Sep 05 19:43:22.544023 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543651    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-etc-sysconfig\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.544023 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543682    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-var-lib-openvswitch\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.544023 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543707    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-host-run-ovn-kubernetes\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.544023 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543753    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0216ba9d-7122-47a3-86eb-8d14ddb478b0-hosts-file\") pod \"node-resolver-5h45d\" (UID: \"0216ba9d-7122-47a3-86eb-8d14ddb478b0\") " pod="openshift-dns/node-resolver-5h45d"
Sep 05 19:43:22.544023 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543733    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9398b13d-ef3a-4d9e-96f4-725c11892e70-registration-dir\") pod \"aws-ebs-csi-driver-node-xnjng\" (UID: \"9398b13d-ef3a-4d9e-96f4-725c11892e70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng"
Sep 05 19:43:22.544023 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543801    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9398b13d-ef3a-4d9e-96f4-725c11892e70-etc-selinux\") pod \"aws-ebs-csi-driver-node-xnjng\" (UID: \"9398b13d-ef3a-4d9e-96f4-725c11892e70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng"
Sep 05 19:43:22.544023 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543807    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9398b13d-ef3a-4d9e-96f4-725c11892e70-registration-dir\") pod \"aws-ebs-csi-driver-node-xnjng\" (UID: \"9398b13d-ef3a-4d9e-96f4-725c11892e70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng"
Sep 05 19:43:22.544023 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543830    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgfpq\" (UniqueName: \"kubernetes.io/projected/0216ba9d-7122-47a3-86eb-8d14ddb478b0-kube-api-access-bgfpq\") pod \"node-resolver-5h45d\" (UID: \"0216ba9d-7122-47a3-86eb-8d14ddb478b0\") " pod="openshift-dns/node-resolver-5h45d"
Sep 05 19:43:22.544023 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543855    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-run\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.544023 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543871    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-etc-sysconfig\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.544023 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543879    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-host-kubelet\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.544023 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543906    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.544023 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543923    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-host-run-ovn-kubernetes\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.545355 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543932    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9398b13d-ef3a-4d9e-96f4-725c11892e70-socket-dir\") pod \"aws-ebs-csi-driver-node-xnjng\" (UID: \"9398b13d-ef3a-4d9e-96f4-725c11892e70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng"
Sep 05 19:43:22.545355 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543957    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-etc-sysctl-conf\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.545355 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543979    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-var-lib-kubelet\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.545355 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544016    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-ovnkube-script-lib\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.545355 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544006    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/334781cf-bbb8-46db-a50c-4d13f6e9f204-cni-binary-copy\") pod \"multus-additional-cni-plugins-wzs5q\" (UID: \"334781cf-bbb8-46db-a50c-4d13f6e9f204\") " pod="openshift-multus/multus-additional-cni-plugins-wzs5q"
Sep 05 19:43:22.545355 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544027    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-run\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.545355 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544042    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9398b13d-ef3a-4d9e-96f4-725c11892e70-etc-selinux\") pod \"aws-ebs-csi-driver-node-xnjng\" (UID: \"9398b13d-ef3a-4d9e-96f4-725c11892e70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng"
Sep 05 19:43:22.545355 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.543955    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/334781cf-bbb8-46db-a50c-4d13f6e9f204-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wzs5q\" (UID: \"334781cf-bbb8-46db-a50c-4d13f6e9f204\") " pod="openshift-multus/multus-additional-cni-plugins-wzs5q"
Sep 05 19:43:22.545355 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544065    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gzqd\" (UniqueName: \"kubernetes.io/projected/27ce30e4-5b20-4020-a55e-c9909e94e228-kube-api-access-2gzqd\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.545355 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544075    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-var-lib-openvswitch\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.545355 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544033    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4d0749ab-114b-4d89-920d-60979f8b485e-iptables-alerter-script\") pod \"iptables-alerter-mhzdb\" (UID: \"4d0749ab-114b-4d89-920d-60979f8b485e\") " pod="openshift-network-operator/iptables-alerter-mhzdb"
Sep 05 19:43:22.545355 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544091    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-env-overrides\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.545355 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544102    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-var-lib-kubelet\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.545355 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544135    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9398b13d-ef3a-4d9e-96f4-725c11892e70-sys-fs\") pod \"aws-ebs-csi-driver-node-xnjng\" (UID: \"9398b13d-ef3a-4d9e-96f4-725c11892e70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng"
Sep 05 19:43:22.545355 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544162    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc4hg\" (UniqueName: \"kubernetes.io/projected/b9720241-ac7b-4d79-9644-233caff66a8c-kube-api-access-fc4hg\") pod \"network-metrics-daemon-rwwbl\" (UID: \"b9720241-ac7b-4d79-9644-233caff66a8c\") " pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:22.545355 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544186    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f4df1d96-03b4-4947-aa18-86dd66548441-konnectivity-ca\") pod \"konnectivity-agent-sdb8j\" (UID: \"f4df1d96-03b4-4947-aa18-86dd66548441\") " pod="kube-system/konnectivity-agent-sdb8j"
Sep 05 19:43:22.545355 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544222    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9398b13d-ef3a-4d9e-96f4-725c11892e70-socket-dir\") pod \"aws-ebs-csi-driver-node-xnjng\" (UID: \"9398b13d-ef3a-4d9e-96f4-725c11892e70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng"
Sep 05 19:43:22.545970 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544249    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/334781cf-bbb8-46db-a50c-4d13f6e9f204-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wzs5q\" (UID: \"334781cf-bbb8-46db-a50c-4d13f6e9f204\") " pod="openshift-multus/multus-additional-cni-plugins-wzs5q"
Sep 05 19:43:22.545970 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544276    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-host-run-netns\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.545970 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544280    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9398b13d-ef3a-4d9e-96f4-725c11892e70-sys-fs\") pod \"aws-ebs-csi-driver-node-xnjng\" (UID: \"9398b13d-ef3a-4d9e-96f4-725c11892e70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng"
Sep 05 19:43:22.545970 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544269    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.545970 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544320    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-host-kubelet\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.545970 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544322    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-etc-sysctl-conf\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.545970 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544419    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-run-openvswitch\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.545970 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544458    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-host-run-netns\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.545970 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544488    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-run-openvswitch\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.545970 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544490    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-etc-modprobe-d\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.545970 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544505    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-env-overrides\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.545970 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544549    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-etc-systemd\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.545970 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544551    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/334781cf-bbb8-46db-a50c-4d13f6e9f204-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wzs5q\" (UID: \"334781cf-bbb8-46db-a50c-4d13f6e9f204\") " pod="openshift-multus/multus-additional-cni-plugins-wzs5q"
Sep 05 19:43:22.545970 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544583    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/334781cf-bbb8-46db-a50c-4d13f6e9f204-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wzs5q\" (UID: \"334781cf-bbb8-46db-a50c-4d13f6e9f204\") " pod="openshift-multus/multus-additional-cni-plugins-wzs5q"
Sep 05 19:43:22.545970 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544593    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-etc-systemd\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.545970 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544591    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-etc-modprobe-d\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.545970 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544611    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-host-cni-netd\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.546548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544636    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59264\" (UniqueName: \"kubernetes.io/projected/9398b13d-ef3a-4d9e-96f4-725c11892e70-kube-api-access-59264\") pod \"aws-ebs-csi-driver-node-xnjng\" (UID: \"9398b13d-ef3a-4d9e-96f4-725c11892e70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng"
Sep 05 19:43:22.546548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544712    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-host-cni-netd\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.546548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544734    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs\") pod \"network-metrics-daemon-rwwbl\" (UID: \"b9720241-ac7b-4d79-9644-233caff66a8c\") " pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:22.546548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544752    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27ce30e4-5b20-4020-a55e-c9909e94e228-tmp\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.546548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544783    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-ovnkube-config\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.546548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544797    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9398b13d-ef3a-4d9e-96f4-725c11892e70-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xnjng\" (UID: \"9398b13d-ef3a-4d9e-96f4-725c11892e70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng"
Sep 05 19:43:22.546548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544822    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq95c\" (UniqueName: \"kubernetes.io/projected/334781cf-bbb8-46db-a50c-4d13f6e9f204-kube-api-access-gq95c\") pod \"multus-additional-cni-plugins-wzs5q\" (UID: \"334781cf-bbb8-46db-a50c-4d13f6e9f204\") " pod="openshift-multus/multus-additional-cni-plugins-wzs5q"
Sep 05 19:43:22.546548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544857    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-host-slash\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.546548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544882    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-host\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.546548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544912    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-systemd-units\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.546548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544954    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-sys\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.546548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544978    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/334781cf-bbb8-46db-a50c-4d13f6e9f204-system-cni-dir\") pod \"multus-additional-cni-plugins-wzs5q\" (UID: \"334781cf-bbb8-46db-a50c-4d13f6e9f204\") " pod="openshift-multus/multus-additional-cni-plugins-wzs5q"
Sep 05 19:43:22.546548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544994    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-host\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.546548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.545003    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9398b13d-ef3a-4d9e-96f4-725c11892e70-device-dir\") pod \"aws-ebs-csi-driver-node-xnjng\" (UID: \"9398b13d-ef3a-4d9e-96f4-725c11892e70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng"
Sep 05 19:43:22.546548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.545056    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9398b13d-ef3a-4d9e-96f4-725c11892e70-device-dir\") pod \"aws-ebs-csi-driver-node-xnjng\" (UID: \"9398b13d-ef3a-4d9e-96f4-725c11892e70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng"
Sep 05 19:43:22.546548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544881    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9398b13d-ef3a-4d9e-96f4-725c11892e70-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xnjng\" (UID: \"9398b13d-ef3a-4d9e-96f4-725c11892e70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng"
Sep 05 19:43:22.546548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.545125    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-systemd-units\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.547095 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.545146    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/334781cf-bbb8-46db-a50c-4d13f6e9f204-cni-binary-copy\") pod \"multus-additional-cni-plugins-wzs5q\" (UID: \"334781cf-bbb8-46db-a50c-4d13f6e9f204\") " pod="openshift-multus/multus-additional-cni-plugins-wzs5q"
Sep 05 19:43:22.547095 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.545172    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-sys\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.547095 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:22.544955    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:22.547095 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.545203    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-host-slash\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.547095 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.545213    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-lib-modules\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.547095 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.545246    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/334781cf-bbb8-46db-a50c-4d13f6e9f204-system-cni-dir\") pod \"multus-additional-cni-plugins-wzs5q\" (UID: \"334781cf-bbb8-46db-a50c-4d13f6e9f204\") " pod="openshift-multus/multus-additional-cni-plugins-wzs5q"
Sep 05 19:43:22.547095 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:22.545260    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs podName:b9720241-ac7b-4d79-9644-233caff66a8c nodeName:}" failed. No retries permitted until 2026-09-05 19:43:23.045235266 +0000 UTC m=+2.074867158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs") pod "network-metrics-daemon-rwwbl" (UID: "b9720241-ac7b-4d79-9644-233caff66a8c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:22.547095 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.544771    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f4df1d96-03b4-4947-aa18-86dd66548441-konnectivity-ca\") pod \"konnectivity-agent-sdb8j\" (UID: \"f4df1d96-03b4-4947-aa18-86dd66548441\") " pod="kube-system/konnectivity-agent-sdb8j"
Sep 05 19:43:22.547095 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.545279    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-etc-openvswitch\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.547095 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.545317    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-lib-modules\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.547095 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.545346    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-ovnkube-config\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.547095 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.545363    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-etc-openvswitch\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.547095 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.545333    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckqjv\" (UniqueName: \"kubernetes.io/projected/63646bb5-7fb0-417d-96a8-24d5c8591bf1-kube-api-access-ckqjv\") pod \"network-check-target-zxsbh\" (UID: \"63646bb5-7fb0-417d-96a8-24d5c8591bf1\") " pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:22.547095 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.545402    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0216ba9d-7122-47a3-86eb-8d14ddb478b0-tmp-dir\") pod \"node-resolver-5h45d\" (UID: \"0216ba9d-7122-47a3-86eb-8d14ddb478b0\") " pod="openshift-dns/node-resolver-5h45d"
Sep 05 19:43:22.547095 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.545428    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-etc-kubernetes\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.547095 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.545454    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f4df1d96-03b4-4947-aa18-86dd66548441-agent-certs\") pod \"konnectivity-agent-sdb8j\" (UID: \"f4df1d96-03b4-4947-aa18-86dd66548441\") " pod="kube-system/konnectivity-agent-sdb8j"
Sep 05 19:43:22.547095 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.545458    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/334781cf-bbb8-46db-a50c-4d13f6e9f204-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wzs5q\" (UID: \"334781cf-bbb8-46db-a50c-4d13f6e9f204\") " pod="openshift-multus/multus-additional-cni-plugins-wzs5q"
Sep 05 19:43:22.547634 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.545478    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d0749ab-114b-4d89-920d-60979f8b485e-host-slash\") pod \"iptables-alerter-mhzdb\" (UID: \"4d0749ab-114b-4d89-920d-60979f8b485e\") " pod="openshift-network-operator/iptables-alerter-mhzdb"
Sep 05 19:43:22.547634 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.545505    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-host-cni-bin\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.547634 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.545528    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbn8f\" (UniqueName: \"kubernetes.io/projected/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-kube-api-access-tbn8f\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.547634 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.545529    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d0749ab-114b-4d89-920d-60979f8b485e-host-slash\") pod \"iptables-alerter-mhzdb\" (UID: \"4d0749ab-114b-4d89-920d-60979f8b485e\") " pod="openshift-network-operator/iptables-alerter-mhzdb"
Sep 05 19:43:22.547634 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.545566    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-host-cni-bin\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.547634 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.545636    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27ce30e4-5b20-4020-a55e-c9909e94e228-etc-kubernetes\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.547634 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.545673    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0216ba9d-7122-47a3-86eb-8d14ddb478b0-tmp-dir\") pod \"node-resolver-5h45d\" (UID: \"0216ba9d-7122-47a3-86eb-8d14ddb478b0\") " pod="openshift-dns/node-resolver-5h45d"
Sep 05 19:43:22.547634 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.546264    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-ovn-node-metrics-cert\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.547634 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.546265    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/27ce30e4-5b20-4020-a55e-c9909e94e228-etc-tuned\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.547634 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.547508    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f4df1d96-03b4-4947-aa18-86dd66548441-agent-certs\") pod \"konnectivity-agent-sdb8j\" (UID: \"f4df1d96-03b4-4947-aa18-86dd66548441\") " pod="kube-system/konnectivity-agent-sdb8j"
Sep 05 19:43:22.547929 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.547718    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27ce30e4-5b20-4020-a55e-c9909e94e228-tmp\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.551401 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.551381    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgvcm\" (UniqueName: \"kubernetes.io/projected/4d0749ab-114b-4d89-920d-60979f8b485e-kube-api-access-bgvcm\") pod \"iptables-alerter-mhzdb\" (UID: \"4d0749ab-114b-4d89-920d-60979f8b485e\") " pod="openshift-network-operator/iptables-alerter-mhzdb"
Sep 05 19:43:22.560035 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.560014    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgfpq\" (UniqueName: \"kubernetes.io/projected/0216ba9d-7122-47a3-86eb-8d14ddb478b0-kube-api-access-bgfpq\") pod \"node-resolver-5h45d\" (UID: \"0216ba9d-7122-47a3-86eb-8d14ddb478b0\") " pod="openshift-dns/node-resolver-5h45d"
Sep 05 19:43:22.567575 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:22.567555    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 19:43:22.567575 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:22.567574    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 19:43:22.567698 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:22.567584    2583 projected.go:196] Error preparing data for projected volume kube-api-access-ckqjv for pod openshift-network-diagnostics/network-check-target-zxsbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:22.567698 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:22.567645    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63646bb5-7fb0-417d-96a8-24d5c8591bf1-kube-api-access-ckqjv podName:63646bb5-7fb0-417d-96a8-24d5c8591bf1 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:23.067622941 +0000 UTC m=+2.097254829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ckqjv" (UniqueName: "kubernetes.io/projected/63646bb5-7fb0-417d-96a8-24d5c8591bf1-kube-api-access-ckqjv") pod "network-check-target-zxsbh" (UID: "63646bb5-7fb0-417d-96a8-24d5c8591bf1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:22.568142 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.568124    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59264\" (UniqueName: \"kubernetes.io/projected/9398b13d-ef3a-4d9e-96f4-725c11892e70-kube-api-access-59264\") pod \"aws-ebs-csi-driver-node-xnjng\" (UID: \"9398b13d-ef3a-4d9e-96f4-725c11892e70\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng"
Sep 05 19:43:22.569104 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.569086    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq95c\" (UniqueName: \"kubernetes.io/projected/334781cf-bbb8-46db-a50c-4d13f6e9f204-kube-api-access-gq95c\") pod \"multus-additional-cni-plugins-wzs5q\" (UID: \"334781cf-bbb8-46db-a50c-4d13f6e9f204\") " pod="openshift-multus/multus-additional-cni-plugins-wzs5q"
Sep 05 19:43:22.572360 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.572344    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gzqd\" (UniqueName: \"kubernetes.io/projected/27ce30e4-5b20-4020-a55e-c9909e94e228-kube-api-access-2gzqd\") pod \"tuned-mpjps\" (UID: \"27ce30e4-5b20-4020-a55e-c9909e94e228\") " pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.572614 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.572596    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc4hg\" (UniqueName: \"kubernetes.io/projected/b9720241-ac7b-4d79-9644-233caff66a8c-kube-api-access-fc4hg\") pod \"network-metrics-daemon-rwwbl\" (UID: \"b9720241-ac7b-4d79-9644-233caff66a8c\") " pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:22.576518 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.576448    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbn8f\" (UniqueName: \"kubernetes.io/projected/f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b-kube-api-access-tbn8f\") pod \"ovnkube-node-mf9tj\" (UID: \"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b\") " pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.668967 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.668895    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-grttc"
Sep 05 19:43:22.679200 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:22.679178    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2809e3e1_0bd8_4cc6_baa9_96ebbb7486f3.slice/crio-8b78d69999eb17a5db1ff861f8d4c8f20cb24d8a4ddc8da9072bc0145e9d3276 WatchSource:0}: Error finding container 8b78d69999eb17a5db1ff861f8d4c8f20cb24d8a4ddc8da9072bc0145e9d3276: Status 404 returned error can't find the container with id 8b78d69999eb17a5db1ff861f8d4c8f20cb24d8a4ddc8da9072bc0145e9d3276
Sep 05 19:43:22.686145 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.686094    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z6kd9"
Sep 05 19:43:22.694104 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:22.694079    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b1f61a_c064_4c08_9bd4_cbe7fe25c934.slice/crio-27e0b3cab508ac8f53c583336e76df27dee67ef4fcf570d177e5e892a9f50856 WatchSource:0}: Error finding container 27e0b3cab508ac8f53c583336e76df27dee67ef4fcf570d177e5e892a9f50856: Status 404 returned error can't find the container with id 27e0b3cab508ac8f53c583336e76df27dee67ef4fcf570d177e5e892a9f50856
Sep 05 19:43:22.704174 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.704157    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mhzdb"
Sep 05 19:43:22.711917 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:22.711894    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0749ab_114b_4d89_920d_60979f8b485e.slice/crio-8c6619075beca1cea506af8b8fbe2fd883169015e2c726621c6772272d8ceccd WatchSource:0}: Error finding container 8c6619075beca1cea506af8b8fbe2fd883169015e2c726621c6772272d8ceccd: Status 404 returned error can't find the container with id 8c6619075beca1cea506af8b8fbe2fd883169015e2c726621c6772272d8ceccd
Sep 05 19:43:22.722517 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.722500    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:22.730491 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:22.730459    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e5cbc4_f09e_417d_a9f9_22fc1b1c239b.slice/crio-e64ebfb0b31d1ec6ac0aea05913e7382b6ad4ecca32e8964caeaf355da6ff273 WatchSource:0}: Error finding container e64ebfb0b31d1ec6ac0aea05913e7382b6ad4ecca32e8964caeaf355da6ff273: Status 404 returned error can't find the container with id e64ebfb0b31d1ec6ac0aea05913e7382b6ad4ecca32e8964caeaf355da6ff273
Sep 05 19:43:22.743503 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.743477    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng"
Sep 05 19:43:22.745744 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.745729    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 19:43:22.751137 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:22.751103    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9398b13d_ef3a_4d9e_96f4_725c11892e70.slice/crio-c356ea50d6c59927ae019e77685aa9934e6cff826fb604bac9d1541d873a0a92 WatchSource:0}: Error finding container c356ea50d6c59927ae019e77685aa9934e6cff826fb604bac9d1541d873a0a92: Status 404 returned error can't find the container with id c356ea50d6c59927ae019e77685aa9934e6cff826fb604bac9d1541d873a0a92
Sep 05 19:43:22.768606 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.768586    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5h45d"
Sep 05 19:43:22.775493 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.775475    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mpjps"
Sep 05 19:43:22.776620 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:22.776600    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0216ba9d_7122_47a3_86eb_8d14ddb478b0.slice/crio-891dd5f262b91ba9b1aa3e219697a50d7469449972305ccf6ad1c21b41818c08 WatchSource:0}: Error finding container 891dd5f262b91ba9b1aa3e219697a50d7469449972305ccf6ad1c21b41818c08: Status 404 returned error can't find the container with id 891dd5f262b91ba9b1aa3e219697a50d7469449972305ccf6ad1c21b41818c08
Sep 05 19:43:22.779557 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.779544    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sdb8j"
Sep 05 19:43:22.784146 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.784125    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wzs5q"
Sep 05 19:43:22.784504 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:22.784484    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ce30e4_5b20_4020_a55e_c9909e94e228.slice/crio-e3a9d400932a1bf3ce16415eeb841b427af25102c62bbdb4da6aa1cd7e853881 WatchSource:0}: Error finding container e3a9d400932a1bf3ce16415eeb841b427af25102c62bbdb4da6aa1cd7e853881: Status 404 returned error can't find the container with id e3a9d400932a1bf3ce16415eeb841b427af25102c62bbdb4da6aa1cd7e853881
Sep 05 19:43:22.789003 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:22.788984    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4df1d96_03b4_4947_aa18_86dd66548441.slice/crio-521a1f983cd0408c8dc6b756e3100913b97fb2dbaf4aef8aad208c468693bf86 WatchSource:0}: Error finding container 521a1f983cd0408c8dc6b756e3100913b97fb2dbaf4aef8aad208c468693bf86: Status 404 returned error can't find the container with id 521a1f983cd0408c8dc6b756e3100913b97fb2dbaf4aef8aad208c468693bf86
Sep 05 19:43:22.793068 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:22.793042    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334781cf_bbb8_46db_a50c_4d13f6e9f204.slice/crio-dedda4a440a66b58d57511cb4c233886ee1591b4d54ccd9f8d8656c98b3df311 WatchSource:0}: Error finding container dedda4a440a66b58d57511cb4c233886ee1591b4d54ccd9f8d8656c98b3df311: Status 404 returned error can't find the container with id dedda4a440a66b58d57511cb4c233886ee1591b4d54ccd9f8d8656c98b3df311
Sep 05 19:43:22.949285 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:22.949164    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83377d0d-0308-4613-bd53-04d2cb4adc3b-original-pull-secret\") pod \"global-pull-secret-syncer-ccth2\" (UID: \"83377d0d-0308-4613-bd53-04d2cb4adc3b\") " pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:22.949438 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:22.949350    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:22.949438 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:22.949409    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83377d0d-0308-4613-bd53-04d2cb4adc3b-original-pull-secret podName:83377d0d-0308-4613-bd53-04d2cb4adc3b nodeName:}" failed. No retries permitted until 2026-09-05 19:43:23.949390483 +0000 UTC m=+2.979022384 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83377d0d-0308-4613-bd53-04d2cb4adc3b-original-pull-secret") pod "global-pull-secret-syncer-ccth2" (UID: "83377d0d-0308-4613-bd53-04d2cb4adc3b") : object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:23.049594 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:23.049567    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs\") pod \"network-metrics-daemon-rwwbl\" (UID: \"b9720241-ac7b-4d79-9644-233caff66a8c\") " pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:23.049793 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:23.049687    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:23.049793 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:23.049747    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs podName:b9720241-ac7b-4d79-9644-233caff66a8c nodeName:}" failed. No retries permitted until 2026-09-05 19:43:24.049727075 +0000 UTC m=+3.079359004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs") pod "network-metrics-daemon-rwwbl" (UID: "b9720241-ac7b-4d79-9644-233caff66a8c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:23.151412 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:23.150764    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckqjv\" (UniqueName: \"kubernetes.io/projected/63646bb5-7fb0-417d-96a8-24d5c8591bf1-kube-api-access-ckqjv\") pod \"network-check-target-zxsbh\" (UID: \"63646bb5-7fb0-417d-96a8-24d5c8591bf1\") " pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:23.151412 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:23.150983    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 19:43:23.151412 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:23.151001    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 19:43:23.151412 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:23.151013    2583 projected.go:196] Error preparing data for projected volume kube-api-access-ckqjv for pod openshift-network-diagnostics/network-check-target-zxsbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:23.151412 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:23.151068    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63646bb5-7fb0-417d-96a8-24d5c8591bf1-kube-api-access-ckqjv podName:63646bb5-7fb0-417d-96a8-24d5c8591bf1 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:24.151051804 +0000 UTC m=+3.180683705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ckqjv" (UniqueName: "kubernetes.io/projected/63646bb5-7fb0-417d-96a8-24d5c8591bf1-kube-api-access-ckqjv") pod "network-check-target-zxsbh" (UID: "63646bb5-7fb0-417d-96a8-24d5c8591bf1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:23.393538 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:23.393410    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 19:38:22 +0000 UTC" deadline="2028-06-03 04:39:59.699147776 +0000 UTC"
Sep 05 19:43:23.393538 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:23.393480    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15272h56m36.305671288s"
Sep 05 19:43:23.467756 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:23.467698    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-grttc" event={"ID":"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3","Type":"ContainerStarted","Data":"8b78d69999eb17a5db1ff861f8d4c8f20cb24d8a4ddc8da9072bc0145e9d3276"}
Sep 05 19:43:23.496632 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:23.496597    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-208.ec2.internal" event={"ID":"ef30841ed7c762f04a60906c21dc14a2","Type":"ContainerStarted","Data":"979676b4adff5ebd2ad94f6f7d91e486cfc3c0d15abc8c02d58f3a4e778dd50d"}
Sep 05 19:43:23.516420 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:23.516385    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-208.ec2.internal" event={"ID":"78a3deb210d787366a6f198baa73b2ab","Type":"ContainerStarted","Data":"0fec068c65d5b98dbcbf1a3a3efb584bfe25b7904467848069748fac2f8f19f9"}
Sep 05 19:43:23.524358 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:23.524325    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sdb8j" event={"ID":"f4df1d96-03b4-4947-aa18-86dd66548441","Type":"ContainerStarted","Data":"521a1f983cd0408c8dc6b756e3100913b97fb2dbaf4aef8aad208c468693bf86"}
Sep 05 19:43:23.544128 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:23.544033    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng" event={"ID":"9398b13d-ef3a-4d9e-96f4-725c11892e70","Type":"ContainerStarted","Data":"c356ea50d6c59927ae019e77685aa9934e6cff826fb604bac9d1541d873a0a92"}
Sep 05 19:43:23.558121 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:23.558076    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj" event={"ID":"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b","Type":"ContainerStarted","Data":"e64ebfb0b31d1ec6ac0aea05913e7382b6ad4ecca32e8964caeaf355da6ff273"}
Sep 05 19:43:23.579843 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:23.579782    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mhzdb" event={"ID":"4d0749ab-114b-4d89-920d-60979f8b485e","Type":"ContainerStarted","Data":"8c6619075beca1cea506af8b8fbe2fd883169015e2c726621c6772272d8ceccd"}
Sep 05 19:43:23.588575 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:23.588543    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z6kd9" event={"ID":"02b1f61a-c064-4c08-9bd4-cbe7fe25c934","Type":"ContainerStarted","Data":"27e0b3cab508ac8f53c583336e76df27dee67ef4fcf570d177e5e892a9f50856"}
Sep 05 19:43:23.595783 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:23.595750    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs5q" event={"ID":"334781cf-bbb8-46db-a50c-4d13f6e9f204","Type":"ContainerStarted","Data":"dedda4a440a66b58d57511cb4c233886ee1591b4d54ccd9f8d8656c98b3df311"}
Sep 05 19:43:23.601368 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:23.601337    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mpjps" event={"ID":"27ce30e4-5b20-4020-a55e-c9909e94e228","Type":"ContainerStarted","Data":"e3a9d400932a1bf3ce16415eeb841b427af25102c62bbdb4da6aa1cd7e853881"}
Sep 05 19:43:23.613886 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:23.613852    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5h45d" event={"ID":"0216ba9d-7122-47a3-86eb-8d14ddb478b0","Type":"ContainerStarted","Data":"891dd5f262b91ba9b1aa3e219697a50d7469449972305ccf6ad1c21b41818c08"}
Sep 05 19:43:23.857292 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:23.857213    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 19:43:23.956411 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:23.956374    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83377d0d-0308-4613-bd53-04d2cb4adc3b-original-pull-secret\") pod \"global-pull-secret-syncer-ccth2\" (UID: \"83377d0d-0308-4613-bd53-04d2cb4adc3b\") " pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:23.956589 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:23.956564    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:23.956653 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:23.956630    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83377d0d-0308-4613-bd53-04d2cb4adc3b-original-pull-secret podName:83377d0d-0308-4613-bd53-04d2cb4adc3b nodeName:}" failed. No retries permitted until 2026-09-05 19:43:25.956609086 +0000 UTC m=+4.986240988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83377d0d-0308-4613-bd53-04d2cb4adc3b-original-pull-secret") pod "global-pull-secret-syncer-ccth2" (UID: "83377d0d-0308-4613-bd53-04d2cb4adc3b") : object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:24.057760 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:24.057720    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs\") pod \"network-metrics-daemon-rwwbl\" (UID: \"b9720241-ac7b-4d79-9644-233caff66a8c\") " pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:24.057944 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:24.057849    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:24.057944 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:24.057917    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs podName:b9720241-ac7b-4d79-9644-233caff66a8c nodeName:}" failed. No retries permitted until 2026-09-05 19:43:26.057898673 +0000 UTC m=+5.087530566 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs") pod "network-metrics-daemon-rwwbl" (UID: "b9720241-ac7b-4d79-9644-233caff66a8c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:24.158493 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:24.158413    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckqjv\" (UniqueName: \"kubernetes.io/projected/63646bb5-7fb0-417d-96a8-24d5c8591bf1-kube-api-access-ckqjv\") pod \"network-check-target-zxsbh\" (UID: \"63646bb5-7fb0-417d-96a8-24d5c8591bf1\") " pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:24.158661 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:24.158581    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 19:43:24.158661 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:24.158604    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 19:43:24.158661 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:24.158643    2583 projected.go:196] Error preparing data for projected volume kube-api-access-ckqjv for pod openshift-network-diagnostics/network-check-target-zxsbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:24.158836 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:24.158773    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63646bb5-7fb0-417d-96a8-24d5c8591bf1-kube-api-access-ckqjv podName:63646bb5-7fb0-417d-96a8-24d5c8591bf1 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:26.15875496 +0000 UTC m=+5.188386870 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ckqjv" (UniqueName: "kubernetes.io/projected/63646bb5-7fb0-417d-96a8-24d5c8591bf1-kube-api-access-ckqjv") pod "network-check-target-zxsbh" (UID: "63646bb5-7fb0-417d-96a8-24d5c8591bf1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:24.395299 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:24.395201    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 19:38:22 +0000 UTC" deadline="2028-02-26 17:59:12.728531192 +0000 UTC"
Sep 05 19:43:24.395299 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:24.395265    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12934h15m48.333270783s"
Sep 05 19:43:24.429196 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:24.428278    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:24.429196 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:24.428417    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccth2" podUID="83377d0d-0308-4613-bd53-04d2cb4adc3b"
Sep 05 19:43:24.429196 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:24.428853    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:24.429196 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:24.428960    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rwwbl" podUID="b9720241-ac7b-4d79-9644-233caff66a8c"
Sep 05 19:43:24.429196 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:24.429019    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:24.429196 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:24.429085    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxsbh" podUID="63646bb5-7fb0-417d-96a8-24d5c8591bf1"
Sep 05 19:43:25.373055 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:25.373021    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 05 19:43:25.970777 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:25.970698    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83377d0d-0308-4613-bd53-04d2cb4adc3b-original-pull-secret\") pod \"global-pull-secret-syncer-ccth2\" (UID: \"83377d0d-0308-4613-bd53-04d2cb4adc3b\") " pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:25.971352 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:25.970892    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:25.971352 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:25.970970    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83377d0d-0308-4613-bd53-04d2cb4adc3b-original-pull-secret podName:83377d0d-0308-4613-bd53-04d2cb4adc3b nodeName:}" failed. No retries permitted until 2026-09-05 19:43:29.970931912 +0000 UTC m=+9.000563814 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83377d0d-0308-4613-bd53-04d2cb4adc3b-original-pull-secret") pod "global-pull-secret-syncer-ccth2" (UID: "83377d0d-0308-4613-bd53-04d2cb4adc3b") : object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:26.071612 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:26.071576    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs\") pod \"network-metrics-daemon-rwwbl\" (UID: \"b9720241-ac7b-4d79-9644-233caff66a8c\") " pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:26.071771 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:26.071737    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:26.071846 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:26.071800    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs podName:b9720241-ac7b-4d79-9644-233caff66a8c nodeName:}" failed. No retries permitted until 2026-09-05 19:43:30.071782195 +0000 UTC m=+9.101414082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs") pod "network-metrics-daemon-rwwbl" (UID: "b9720241-ac7b-4d79-9644-233caff66a8c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:26.172743 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:26.172706    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckqjv\" (UniqueName: \"kubernetes.io/projected/63646bb5-7fb0-417d-96a8-24d5c8591bf1-kube-api-access-ckqjv\") pod \"network-check-target-zxsbh\" (UID: \"63646bb5-7fb0-417d-96a8-24d5c8591bf1\") " pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:26.172915 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:26.172855    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 19:43:26.172915 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:26.172872    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 19:43:26.172915 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:26.172885    2583 projected.go:196] Error preparing data for projected volume kube-api-access-ckqjv for pod openshift-network-diagnostics/network-check-target-zxsbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:26.173078 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:26.172939    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63646bb5-7fb0-417d-96a8-24d5c8591bf1-kube-api-access-ckqjv podName:63646bb5-7fb0-417d-96a8-24d5c8591bf1 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:30.172921428 +0000 UTC m=+9.202553329 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ckqjv" (UniqueName: "kubernetes.io/projected/63646bb5-7fb0-417d-96a8-24d5c8591bf1-kube-api-access-ckqjv") pod "network-check-target-zxsbh" (UID: "63646bb5-7fb0-417d-96a8-24d5c8591bf1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:26.428635 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:26.428606    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:26.428772 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:26.428730    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccth2" podUID="83377d0d-0308-4613-bd53-04d2cb4adc3b"
Sep 05 19:43:26.428772 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:26.428757    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:26.428898 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:26.428875    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rwwbl" podUID="b9720241-ac7b-4d79-9644-233caff66a8c"
Sep 05 19:43:26.428950 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:26.428935    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:26.429044 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:26.429002    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxsbh" podUID="63646bb5-7fb0-417d-96a8-24d5c8591bf1"
Sep 05 19:43:28.429586 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:28.429031    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:28.429586 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:28.429067    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:28.429586 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:28.429249    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxsbh" podUID="63646bb5-7fb0-417d-96a8-24d5c8591bf1"
Sep 05 19:43:28.429586 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:28.429314    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccth2" podUID="83377d0d-0308-4613-bd53-04d2cb4adc3b"
Sep 05 19:43:28.429586 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:28.429370    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:28.429586 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:28.429454    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rwwbl" podUID="b9720241-ac7b-4d79-9644-233caff66a8c"
Sep 05 19:43:30.008583 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:30.008531    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83377d0d-0308-4613-bd53-04d2cb4adc3b-original-pull-secret\") pod \"global-pull-secret-syncer-ccth2\" (UID: \"83377d0d-0308-4613-bd53-04d2cb4adc3b\") " pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:30.008961 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:30.008753    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:30.008961 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:30.008810    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83377d0d-0308-4613-bd53-04d2cb4adc3b-original-pull-secret podName:83377d0d-0308-4613-bd53-04d2cb4adc3b nodeName:}" failed. No retries permitted until 2026-09-05 19:43:38.008797126 +0000 UTC m=+17.038429008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83377d0d-0308-4613-bd53-04d2cb4adc3b-original-pull-secret") pod "global-pull-secret-syncer-ccth2" (UID: "83377d0d-0308-4613-bd53-04d2cb4adc3b") : object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:30.109745 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:30.109702    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs\") pod \"network-metrics-daemon-rwwbl\" (UID: \"b9720241-ac7b-4d79-9644-233caff66a8c\") " pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:30.109914 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:30.109889    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:30.109983 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:30.109970    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs podName:b9720241-ac7b-4d79-9644-233caff66a8c nodeName:}" failed. No retries permitted until 2026-09-05 19:43:38.109949395 +0000 UTC m=+17.139581298 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs") pod "network-metrics-daemon-rwwbl" (UID: "b9720241-ac7b-4d79-9644-233caff66a8c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:30.210370 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:30.210333    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckqjv\" (UniqueName: \"kubernetes.io/projected/63646bb5-7fb0-417d-96a8-24d5c8591bf1-kube-api-access-ckqjv\") pod \"network-check-target-zxsbh\" (UID: \"63646bb5-7fb0-417d-96a8-24d5c8591bf1\") " pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:30.210549 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:30.210526    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 19:43:30.210549 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:30.210548    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 19:43:30.210658 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:30.210563    2583 projected.go:196] Error preparing data for projected volume kube-api-access-ckqjv for pod openshift-network-diagnostics/network-check-target-zxsbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:30.210658 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:30.210621    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63646bb5-7fb0-417d-96a8-24d5c8591bf1-kube-api-access-ckqjv podName:63646bb5-7fb0-417d-96a8-24d5c8591bf1 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:38.210604278 +0000 UTC m=+17.240236176 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ckqjv" (UniqueName: "kubernetes.io/projected/63646bb5-7fb0-417d-96a8-24d5c8591bf1-kube-api-access-ckqjv") pod "network-check-target-zxsbh" (UID: "63646bb5-7fb0-417d-96a8-24d5c8591bf1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:30.428760 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:30.428668    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:30.428916 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:30.428880    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rwwbl" podUID="b9720241-ac7b-4d79-9644-233caff66a8c"
Sep 05 19:43:30.429302 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:30.429279    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:30.429426 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:30.429376    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxsbh" podUID="63646bb5-7fb0-417d-96a8-24d5c8591bf1"
Sep 05 19:43:30.429426 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:30.429421    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:30.429533 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:30.429486    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccth2" podUID="83377d0d-0308-4613-bd53-04d2cb4adc3b"
Sep 05 19:43:32.429024 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:32.428982    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:32.429571 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:32.429038    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:32.429571 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:32.429098    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:32.429571 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:32.429130    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccth2" podUID="83377d0d-0308-4613-bd53-04d2cb4adc3b"
Sep 05 19:43:32.429571 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:32.429204    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxsbh" podUID="63646bb5-7fb0-417d-96a8-24d5c8591bf1"
Sep 05 19:43:32.429571 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:32.429306    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rwwbl" podUID="b9720241-ac7b-4d79-9644-233caff66a8c"
Sep 05 19:43:34.428768 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:34.428732    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:34.429252 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:34.428735    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:34.429252 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:34.428849    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxsbh" podUID="63646bb5-7fb0-417d-96a8-24d5c8591bf1"
Sep 05 19:43:34.429252 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:34.428740    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:34.429252 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:34.428942    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rwwbl" podUID="b9720241-ac7b-4d79-9644-233caff66a8c"
Sep 05 19:43:34.429252 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:34.429034    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccth2" podUID="83377d0d-0308-4613-bd53-04d2cb4adc3b"
Sep 05 19:43:36.428511 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:36.428477    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:36.428511 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:36.428515    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:36.428910 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:36.428518    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:36.428910 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:36.428603    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxsbh" podUID="63646bb5-7fb0-417d-96a8-24d5c8591bf1"
Sep 05 19:43:36.428910 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:36.428733    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rwwbl" podUID="b9720241-ac7b-4d79-9644-233caff66a8c"
Sep 05 19:43:36.428910 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:36.428803    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccth2" podUID="83377d0d-0308-4613-bd53-04d2cb4adc3b"
Sep 05 19:43:38.065414 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:38.065374    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83377d0d-0308-4613-bd53-04d2cb4adc3b-original-pull-secret\") pod \"global-pull-secret-syncer-ccth2\" (UID: \"83377d0d-0308-4613-bd53-04d2cb4adc3b\") " pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:38.065958 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:38.065493    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:38.065958 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:38.065552    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83377d0d-0308-4613-bd53-04d2cb4adc3b-original-pull-secret podName:83377d0d-0308-4613-bd53-04d2cb4adc3b nodeName:}" failed. No retries permitted until 2026-09-05 19:43:54.06553829 +0000 UTC m=+33.095170173 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83377d0d-0308-4613-bd53-04d2cb4adc3b-original-pull-secret") pod "global-pull-secret-syncer-ccth2" (UID: "83377d0d-0308-4613-bd53-04d2cb4adc3b") : object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:38.166513 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:38.166475    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs\") pod \"network-metrics-daemon-rwwbl\" (UID: \"b9720241-ac7b-4d79-9644-233caff66a8c\") " pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:38.166670 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:38.166639    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:38.166720 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:38.166707    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs podName:b9720241-ac7b-4d79-9644-233caff66a8c nodeName:}" failed. No retries permitted until 2026-09-05 19:43:54.166692119 +0000 UTC m=+33.196324002 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs") pod "network-metrics-daemon-rwwbl" (UID: "b9720241-ac7b-4d79-9644-233caff66a8c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:38.267456 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:38.267401    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckqjv\" (UniqueName: \"kubernetes.io/projected/63646bb5-7fb0-417d-96a8-24d5c8591bf1-kube-api-access-ckqjv\") pod \"network-check-target-zxsbh\" (UID: \"63646bb5-7fb0-417d-96a8-24d5c8591bf1\") " pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:38.267635 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:38.267581    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 19:43:38.267635 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:38.267603    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 19:43:38.267635 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:38.267615    2583 projected.go:196] Error preparing data for projected volume kube-api-access-ckqjv for pod openshift-network-diagnostics/network-check-target-zxsbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:38.267774 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:38.267667    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63646bb5-7fb0-417d-96a8-24d5c8591bf1-kube-api-access-ckqjv podName:63646bb5-7fb0-417d-96a8-24d5c8591bf1 nodeName:}" failed. No retries permitted until 2026-09-05 19:43:54.267653549 +0000 UTC m=+33.297285443 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ckqjv" (UniqueName: "kubernetes.io/projected/63646bb5-7fb0-417d-96a8-24d5c8591bf1-kube-api-access-ckqjv") pod "network-check-target-zxsbh" (UID: "63646bb5-7fb0-417d-96a8-24d5c8591bf1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:38.428925 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:38.428839    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:38.429074 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:38.428964    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:38.429074 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:38.428977    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rwwbl" podUID="b9720241-ac7b-4d79-9644-233caff66a8c"
Sep 05 19:43:38.429264 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:38.429069    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxsbh" podUID="63646bb5-7fb0-417d-96a8-24d5c8591bf1"
Sep 05 19:43:38.429264 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:38.429076    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:38.429264 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:38.429171    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccth2" podUID="83377d0d-0308-4613-bd53-04d2cb4adc3b"
Sep 05 19:43:40.428911 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:40.428876    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:40.429387 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:40.428876    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:40.429387 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:40.429019    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccth2" podUID="83377d0d-0308-4613-bd53-04d2cb4adc3b"
Sep 05 19:43:40.429387 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:40.429147    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rwwbl" podUID="b9720241-ac7b-4d79-9644-233caff66a8c"
Sep 05 19:43:40.429387 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:40.428876    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:40.429387 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:40.429254    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxsbh" podUID="63646bb5-7fb0-417d-96a8-24d5c8591bf1"
Sep 05 19:43:41.672033 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:41.671826    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-208.ec2.internal" event={"ID":"ef30841ed7c762f04a60906c21dc14a2","Type":"ContainerStarted","Data":"c92b4c956742c00ca7940527c35ffe59bddf3ed8b7704a06d737305f751a550c"}
Sep 05 19:43:41.689013 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:41.688920    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-208.ec2.internal" podStartSLOduration=19.688899984 podStartE2EDuration="19.688899984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:41.688265423 +0000 UTC m=+20.717897326" watchObservedRunningTime="2026-09-05 19:43:41.688899984 +0000 UTC m=+20.718531891"
Sep 05 19:43:42.429444 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.429218    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:42.429568 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.429219    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:42.429568 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:42.429481    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxsbh" podUID="63646bb5-7fb0-417d-96a8-24d5c8591bf1"
Sep 05 19:43:42.429568 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.429219    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:42.429568 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:42.429548    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rwwbl" podUID="b9720241-ac7b-4d79-9644-233caff66a8c"
Sep 05 19:43:42.429722 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:42.429618    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccth2" podUID="83377d0d-0308-4613-bd53-04d2cb4adc3b"
Sep 05 19:43:42.675955 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.675907    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5h45d" event={"ID":"0216ba9d-7122-47a3-86eb-8d14ddb478b0","Type":"ContainerStarted","Data":"a2aa11127f09c7187e2b2d98c7d00d0da619908e71b7e8fca09d81f5d0e0c1d6"}
Sep 05 19:43:42.678389 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.678353    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-grttc" event={"ID":"2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3","Type":"ContainerStarted","Data":"3f8b9ae6749e4063606e6a66d57e4acd88b48e1fc97c78076d4d345b3564f2b6"}
Sep 05 19:43:42.681845 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.681775    2583 generic.go:356] "Generic (PLEG): container finished" podID="78a3deb210d787366a6f198baa73b2ab" containerID="bb6421f310518998cc4b7d0ee75643f934df1e06c5483eafabdd4d366e05b398" exitCode=0
Sep 05 19:43:42.681961 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.681869    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-208.ec2.internal" event={"ID":"78a3deb210d787366a6f198baa73b2ab","Type":"ContainerDied","Data":"bb6421f310518998cc4b7d0ee75643f934df1e06c5483eafabdd4d366e05b398"}
Sep 05 19:43:42.683899 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.683613    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sdb8j" event={"ID":"f4df1d96-03b4-4947-aa18-86dd66548441","Type":"ContainerStarted","Data":"cf79f1f45cb85d91bdb52876a7b1d091c08a0022f5d94018e50d550a3596e591"}
Sep 05 19:43:42.685411 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.685386    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng" event={"ID":"9398b13d-ef3a-4d9e-96f4-725c11892e70","Type":"ContainerStarted","Data":"9fdde77d5d71a2a6af2a3be544ce005c751a79bd8d70e0363961bb4d6e940b1c"}
Sep 05 19:43:42.688539 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.688462    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf9tj_f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b/ovn-acl-logging/0.log"
Sep 05 19:43:42.688942 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.688919    2583 generic.go:356] "Generic (PLEG): container finished" podID="f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b" containerID="13002a71ed8bbad7ac6da2c075f276952eb39ccc6269ab7bd6fc3d1ad73ca138" exitCode=1
Sep 05 19:43:42.689103 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.689001    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj" event={"ID":"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b","Type":"ContainerStarted","Data":"b48fa712fa7d86eba990bccbb1035b587c2e3d0f2d596b06af8404dfa0c7346a"}
Sep 05 19:43:42.689243 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.689227    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj" event={"ID":"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b","Type":"ContainerStarted","Data":"4bf8fcf9f51a16deb9fc95984ed4532d7842a6d5e59087ccc37a1dd977f14040"}
Sep 05 19:43:42.689331 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.689315    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj" event={"ID":"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b","Type":"ContainerStarted","Data":"237a234cbab80584da325a1e206c564ef8334a252bd666638520c7434d3f8d07"}
Sep 05 19:43:42.689393 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.689366    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj" event={"ID":"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b","Type":"ContainerStarted","Data":"2db9f01f90def44629c13a05045446eff7b638bc04e864fa259bb9103caeae64"}
Sep 05 19:43:42.689478 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.689461    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj" event={"ID":"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b","Type":"ContainerDied","Data":"13002a71ed8bbad7ac6da2c075f276952eb39ccc6269ab7bd6fc3d1ad73ca138"}
Sep 05 19:43:42.689527 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.689517    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj" event={"ID":"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b","Type":"ContainerStarted","Data":"0bb282e4e4b8770b6a4212accd6675a0bfae952f8b70419cd589778e89a98995"}
Sep 05 19:43:42.690634 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.690612    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z6kd9" event={"ID":"02b1f61a-c064-4c08-9bd4-cbe7fe25c934","Type":"ContainerStarted","Data":"5014d4a97bec60307be818546249ba3a4753f2bc900addb4bf2cb0b35c9cae0e"}
Sep 05 19:43:42.692326 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.692292    2583 generic.go:356] "Generic (PLEG): container finished" podID="334781cf-bbb8-46db-a50c-4d13f6e9f204" containerID="490fc7cf1944671af8cab7dd7437af9ae481171d03b465f9e0826739c47a92b3" exitCode=0
Sep 05 19:43:42.692419 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.692383    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs5q" event={"ID":"334781cf-bbb8-46db-a50c-4d13f6e9f204","Type":"ContainerDied","Data":"490fc7cf1944671af8cab7dd7437af9ae481171d03b465f9e0826739c47a92b3"}
Sep 05 19:43:42.692885 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.692842    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5h45d" podStartSLOduration=7.284307924 podStartE2EDuration="21.692825871s" totalImagesPullingTime="14.408517947s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:42.692323217 +0000 UTC m=+21.721955124" watchObservedRunningTime="2026-09-05 19:43:42.692825871 +0000 UTC m=+21.722457777"
Sep 05 19:43:42.695517 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.694857    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mpjps" event={"ID":"27ce30e4-5b20-4020-a55e-c9909e94e228","Type":"ContainerStarted","Data":"fdb1bffe569bf33f8ed3954812f1a039eba2b698911253db03338ab6a15deb7e"}
Sep 05 19:43:42.740025 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.739986    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-grttc" podStartSLOduration=2.602011162 podStartE2EDuration="21.739968034s" totalImagesPullingTime="19.137956872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:42.725844462 +0000 UTC m=+21.755476367" watchObservedRunningTime="2026-09-05 19:43:42.739968034 +0000 UTC m=+21.769599940"
Sep 05 19:43:42.740276 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.740248    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z6kd9" podStartSLOduration=3.08395958 podStartE2EDuration="21.740241385s" totalImagesPullingTime="18.656281805s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:42.739894492 +0000 UTC m=+21.769526396" watchObservedRunningTime="2026-09-05 19:43:42.740241385 +0000 UTC m=+21.769873292"
Sep 05 19:43:42.756555 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.756518    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sdb8j" podStartSLOduration=3.144914226 podStartE2EDuration="21.756502124s" totalImagesPullingTime="18.611587898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:42.756314867 +0000 UTC m=+21.785946763" watchObservedRunningTime="2026-09-05 19:43:42.756502124 +0000 UTC m=+21.786134030"
Sep 05 19:43:42.802639 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:42.802597    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mpjps" podStartSLOduration=3.183937562 podStartE2EDuration="21.802582055s" totalImagesPullingTime="18.618644493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:42.802252937 +0000 UTC m=+21.831884843" watchObservedRunningTime="2026-09-05 19:43:42.802582055 +0000 UTC m=+21.832213961"
Sep 05 19:43:43.110000 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:43.109855    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 05 19:43:43.402626 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:43.402408    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-05T19:43:43.109874375Z","UUID":"33aabd01-e5ad-43af-b477-b39e55dc994a","Handler":null,"Name":"","Endpoint":""}
Sep 05 19:43:43.403689 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:43.403669    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 05 19:43:43.403798 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:43.403699    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 05 19:43:43.698930 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:43.698850    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng" event={"ID":"9398b13d-ef3a-4d9e-96f4-725c11892e70","Type":"ContainerStarted","Data":"b75c4bb14093862986a83284cde8d8ae296b6cc881d74f516fdcbeb7a7bb3d6a"}
Sep 05 19:43:43.700739 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:43.700712    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mhzdb" event={"ID":"4d0749ab-114b-4d89-920d-60979f8b485e","Type":"ContainerStarted","Data":"17c618c236f7f4b3be9b2749409b9284f771795d83444e2d28a84817d2561054"}
Sep 05 19:43:43.703315 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:43.703252    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-208.ec2.internal" event={"ID":"78a3deb210d787366a6f198baa73b2ab","Type":"ContainerStarted","Data":"e5228f5497381a43792362b5d37376effa201690730226eb70878797c95d747e"}
Sep 05 19:43:43.717929 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:43.717889    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mhzdb" podStartSLOduration=4.04929085 podStartE2EDuration="22.717873944s" totalImagesPullingTime="18.668583094s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:43.717525576 +0000 UTC m=+22.747157482" watchObservedRunningTime="2026-09-05 19:43:43.717873944 +0000 UTC m=+22.747505848"
Sep 05 19:43:43.734780 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:43.734729    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-208.ec2.internal" podStartSLOduration=21.734711284 podStartE2EDuration="21.734711284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:43.734084647 +0000 UTC m=+22.763716553" watchObservedRunningTime="2026-09-05 19:43:43.734711284 +0000 UTC m=+22.764343190"
Sep 05 19:43:44.429443 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:44.429253    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:44.429624 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:44.429302    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:44.429624 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:44.429538    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxsbh" podUID="63646bb5-7fb0-417d-96a8-24d5c8591bf1"
Sep 05 19:43:44.429624 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:44.429313    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:44.429793 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:44.429639    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rwwbl" podUID="b9720241-ac7b-4d79-9644-233caff66a8c"
Sep 05 19:43:44.429793 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:44.429692    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccth2" podUID="83377d0d-0308-4613-bd53-04d2cb4adc3b"
Sep 05 19:43:44.707503 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:44.707420    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng" event={"ID":"9398b13d-ef3a-4d9e-96f4-725c11892e70","Type":"ContainerStarted","Data":"00c7eeab5469b43854379fb90a3d983fb1789b44a4da5ab28a483065e719a550"}
Sep 05 19:43:44.710499 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:44.710480    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf9tj_f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b/ovn-acl-logging/0.log"
Sep 05 19:43:44.710947 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:44.710914    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj" event={"ID":"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b","Type":"ContainerStarted","Data":"42eedb939c0502f5cc19e90a968e2f910abdb7a0ccbdaea98996b5478f6d24b3"}
Sep 05 19:43:44.726911 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:44.726872    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xnjng" podStartSLOduration=3.4442248429999998 podStartE2EDuration="23.726860413s" totalImagesPullingTime="20.28263557s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:44.72679051 +0000 UTC m=+23.756422418" watchObservedRunningTime="2026-09-05 19:43:44.726860413 +0000 UTC m=+23.756492318"
Sep 05 19:43:46.428264 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:46.428230    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:46.428264 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:46.428252    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:46.428702 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:46.428240    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:46.428702 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:46.428348    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccth2" podUID="83377d0d-0308-4613-bd53-04d2cb4adc3b"
Sep 05 19:43:46.428702 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:46.428393    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxsbh" podUID="63646bb5-7fb0-417d-96a8-24d5c8591bf1"
Sep 05 19:43:46.428702 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:46.428473    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rwwbl" podUID="b9720241-ac7b-4d79-9644-233caff66a8c"
Sep 05 19:43:47.503491 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:47.503464    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sdb8j"
Sep 05 19:43:47.503983 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:47.503966    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sdb8j"
Sep 05 19:43:47.721039 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:47.720976    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf9tj_f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b/ovn-acl-logging/0.log"
Sep 05 19:43:47.721420 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:47.721395    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj" event={"ID":"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b","Type":"ContainerStarted","Data":"5d1c967fb5cf46855a2be2b65d928b88c88be8610a35fcb94b0fb1ca2c5d3ebb"}
Sep 05 19:43:47.721639 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:47.721622    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:47.721851 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:47.721834    2583 scope.go:117] "RemoveContainer" containerID="13002a71ed8bbad7ac6da2c075f276952eb39ccc6269ab7bd6fc3d1ad73ca138"
Sep 05 19:43:47.723308 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:47.723288    2583 generic.go:356] "Generic (PLEG): container finished" podID="334781cf-bbb8-46db-a50c-4d13f6e9f204" containerID="945391f8425a85868633133c3eda52ad3e06d5ea3eab8dc5d3140a3e2b22c983" exitCode=0
Sep 05 19:43:47.723404 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:47.723367    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs5q" event={"ID":"334781cf-bbb8-46db-a50c-4d13f6e9f204","Type":"ContainerDied","Data":"945391f8425a85868633133c3eda52ad3e06d5ea3eab8dc5d3140a3e2b22c983"}
Sep 05 19:43:47.723589 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:47.723568    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sdb8j"
Sep 05 19:43:47.724047 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:47.724033    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sdb8j"
Sep 05 19:43:47.738226 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:47.738206    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:48.428668 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:48.428631    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:48.428927 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:48.428748    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:48.428927 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:48.428783    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:48.428927 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:48.428771    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccth2" podUID="83377d0d-0308-4613-bd53-04d2cb4adc3b"
Sep 05 19:43:48.428927 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:48.428909    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxsbh" podUID="63646bb5-7fb0-417d-96a8-24d5c8591bf1"
Sep 05 19:43:48.429163 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:48.429060    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rwwbl" podUID="b9720241-ac7b-4d79-9644-233caff66a8c"
Sep 05 19:43:48.729720 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:48.729657    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf9tj_f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b/ovn-acl-logging/0.log"
Sep 05 19:43:48.730425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:48.730055    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj" event={"ID":"f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b","Type":"ContainerStarted","Data":"5fcf9e1dcc3ea674ebd31dc75b235ba491f6cc51041fac69a9442717a2bda409"}
Sep 05 19:43:48.730425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:48.730095    2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 05 19:43:48.730425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:48.730330    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:48.744741 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:48.744719    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:48.790950 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:48.790902    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj" podStartSLOduration=9.105062058 podStartE2EDuration="27.790885479s" totalImagesPullingTime="18.685823421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:48.762482227 +0000 UTC m=+27.792114135" watchObservedRunningTime="2026-09-05 19:43:48.790885479 +0000 UTC m=+27.820517417"
Sep 05 19:43:49.148061 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:49.148026    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ccth2"]
Sep 05 19:43:49.148226 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:49.148168    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:49.148319 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:49.148283    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccth2" podUID="83377d0d-0308-4613-bd53-04d2cb4adc3b"
Sep 05 19:43:49.152709 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:49.152686    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rwwbl"]
Sep 05 19:43:49.152809 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:49.152778    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:49.152892 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:49.152873    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rwwbl" podUID="b9720241-ac7b-4d79-9644-233caff66a8c"
Sep 05 19:43:49.153319 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:49.153301    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zxsbh"]
Sep 05 19:43:49.153393 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:49.153379    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:49.153481 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:49.153465    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxsbh" podUID="63646bb5-7fb0-417d-96a8-24d5c8591bf1"
Sep 05 19:43:49.733951 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:49.733916    2583 generic.go:356] "Generic (PLEG): container finished" podID="334781cf-bbb8-46db-a50c-4d13f6e9f204" containerID="a61dfc35c5622a0e577e6eac80d2e49419e0b2246f12b26c5e488c2a945d30c6" exitCode=0
Sep 05 19:43:49.734646 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:49.734007    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs5q" event={"ID":"334781cf-bbb8-46db-a50c-4d13f6e9f204","Type":"ContainerDied","Data":"a61dfc35c5622a0e577e6eac80d2e49419e0b2246f12b26c5e488c2a945d30c6"}
Sep 05 19:43:49.734646 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:49.734051    2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 05 19:43:50.429137 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:50.428924    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:50.429304 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:50.428923    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:50.429304 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:50.429221    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxsbh" podUID="63646bb5-7fb0-417d-96a8-24d5c8591bf1"
Sep 05 19:43:50.429420 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:50.429316    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccth2" podUID="83377d0d-0308-4613-bd53-04d2cb4adc3b"
Sep 05 19:43:50.736314 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:50.736281    2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 05 19:43:51.157031 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:51.156998    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:43:51.429685 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:51.429616    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:51.429819 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:51.429734    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rwwbl" podUID="b9720241-ac7b-4d79-9644-233caff66a8c"
Sep 05 19:43:51.741213 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:51.741128    2583 generic.go:356] "Generic (PLEG): container finished" podID="334781cf-bbb8-46db-a50c-4d13f6e9f204" containerID="e558132a117aaa68c9888f4cc5c0cb9b1451f1041b8db2bc565ad5bbbfe57bd1" exitCode=0
Sep 05 19:43:51.741644 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:51.741203    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs5q" event={"ID":"334781cf-bbb8-46db-a50c-4d13f6e9f204","Type":"ContainerDied","Data":"e558132a117aaa68c9888f4cc5c0cb9b1451f1041b8db2bc565ad5bbbfe57bd1"}
Sep 05 19:43:52.428653 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:52.428624    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:52.428842 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:52.428629    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:52.428842 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:52.428731    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxsbh" podUID="63646bb5-7fb0-417d-96a8-24d5c8591bf1"
Sep 05 19:43:52.428842 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:52.428832    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccth2" podUID="83377d0d-0308-4613-bd53-04d2cb4adc3b"
Sep 05 19:43:53.429315 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:53.429285    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:53.429810 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:53.429407    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rwwbl" podUID="b9720241-ac7b-4d79-9644-233caff66a8c"
Sep 05 19:43:54.087568 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.087529    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83377d0d-0308-4613-bd53-04d2cb4adc3b-original-pull-secret\") pod \"global-pull-secret-syncer-ccth2\" (UID: \"83377d0d-0308-4613-bd53-04d2cb4adc3b\") " pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:54.087749 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:54.087688    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:54.087815 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:54.087764    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83377d0d-0308-4613-bd53-04d2cb4adc3b-original-pull-secret podName:83377d0d-0308-4613-bd53-04d2cb4adc3b nodeName:}" failed. No retries permitted until 2026-09-05 19:44:26.087742077 +0000 UTC m=+65.117373961 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/83377d0d-0308-4613-bd53-04d2cb4adc3b-original-pull-secret") pod "global-pull-secret-syncer-ccth2" (UID: "83377d0d-0308-4613-bd53-04d2cb4adc3b") : object "kube-system"/"original-pull-secret" not registered
Sep 05 19:43:54.188176 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.188143    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs\") pod \"network-metrics-daemon-rwwbl\" (UID: \"b9720241-ac7b-4d79-9644-233caff66a8c\") " pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:54.188332 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:54.188244    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:54.188332 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:54.188308    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs podName:b9720241-ac7b-4d79-9644-233caff66a8c nodeName:}" failed. No retries permitted until 2026-09-05 19:44:26.188282093 +0000 UTC m=+65.217913976 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs") pod "network-metrics-daemon-rwwbl" (UID: "b9720241-ac7b-4d79-9644-233caff66a8c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 05 19:43:54.288800 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.288771    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckqjv\" (UniqueName: \"kubernetes.io/projected/63646bb5-7fb0-417d-96a8-24d5c8591bf1-kube-api-access-ckqjv\") pod \"network-check-target-zxsbh\" (UID: \"63646bb5-7fb0-417d-96a8-24d5c8591bf1\") " pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:54.288968 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:54.288947    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 05 19:43:54.289031 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:54.288975    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 05 19:43:54.289031 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:54.288989    2583 projected.go:196] Error preparing data for projected volume kube-api-access-ckqjv for pod openshift-network-diagnostics/network-check-target-zxsbh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:54.289162 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:54.289041    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/63646bb5-7fb0-417d-96a8-24d5c8591bf1-kube-api-access-ckqjv podName:63646bb5-7fb0-417d-96a8-24d5c8591bf1 nodeName:}" failed. No retries permitted until 2026-09-05 19:44:26.28902862 +0000 UTC m=+65.318660516 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ckqjv" (UniqueName: "kubernetes.io/projected/63646bb5-7fb0-417d-96a8-24d5c8591bf1-kube-api-access-ckqjv") pod "network-check-target-zxsbh" (UID: "63646bb5-7fb0-417d-96a8-24d5c8591bf1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 05 19:43:54.429079 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.429044    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:54.429262 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.429053    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:54.429262 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:54.429187    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zxsbh" podUID="63646bb5-7fb0-417d-96a8-24d5c8591bf1"
Sep 05 19:43:54.429262 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:54.429253    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ccth2" podUID="83377d0d-0308-4613-bd53-04d2cb4adc3b"
Sep 05 19:43:54.745358 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.745277    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-208.ec2.internal" event="NodeReady"
Sep 05 19:43:54.745573 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.745401    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 05 19:43:54.796763 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.796730    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zj8hk"]
Sep 05 19:43:54.821938 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.821907    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6ls6n"]
Sep 05 19:43:54.822148 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.822008    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:43:54.825359 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.825330    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 05 19:43:54.825502 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.825484    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 05 19:43:54.825574 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.825550    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6x9hn\""
Sep 05 19:43:54.835988 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.835967    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6ls6n"]
Sep 05 19:43:54.836099 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.836002    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zj8hk"]
Sep 05 19:43:54.836099 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.836033    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6ls6n"
Sep 05 19:43:54.838443 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.838421    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h7tzm\""
Sep 05 19:43:54.838558 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.838455    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 05 19:43:54.838558 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.838427    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 05 19:43:54.838558 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.838424    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 05 19:43:54.893725 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.893688    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-tmp-dir\") pod \"dns-default-zj8hk\" (UID: \"b1df5064-2ffa-4312-b6f3-cd6dd02c156a\") " pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:43:54.893939 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.893735    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p8jd\" (UniqueName: \"kubernetes.io/projected/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-kube-api-access-2p8jd\") pod \"dns-default-zj8hk\" (UID: \"b1df5064-2ffa-4312-b6f3-cd6dd02c156a\") " pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:43:54.893939 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.893889    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls\") pod \"dns-default-zj8hk\" (UID: \"b1df5064-2ffa-4312-b6f3-cd6dd02c156a\") " pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:43:54.893939 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.893934    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-config-volume\") pod \"dns-default-zj8hk\" (UID: \"b1df5064-2ffa-4312-b6f3-cd6dd02c156a\") " pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:43:54.994635 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.994597    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls\") pod \"dns-default-zj8hk\" (UID: \"b1df5064-2ffa-4312-b6f3-cd6dd02c156a\") " pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:43:54.994827 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:54.994729    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 19:43:54.994827 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.994737    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-config-volume\") pod \"dns-default-zj8hk\" (UID: \"b1df5064-2ffa-4312-b6f3-cd6dd02c156a\") " pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:43:54.994827 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:54.994809    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls podName:b1df5064-2ffa-4312-b6f3-cd6dd02c156a nodeName:}" failed. No retries permitted until 2026-09-05 19:43:55.494773908 +0000 UTC m=+34.524405807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls") pod "dns-default-zj8hk" (UID: "b1df5064-2ffa-4312-b6f3-cd6dd02c156a") : secret "dns-default-metrics-tls" not found
Sep 05 19:43:54.994998 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.994829    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-tmp-dir\") pod \"dns-default-zj8hk\" (UID: \"b1df5064-2ffa-4312-b6f3-cd6dd02c156a\") " pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:43:54.994998 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.994864    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p8jd\" (UniqueName: \"kubernetes.io/projected/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-kube-api-access-2p8jd\") pod \"dns-default-zj8hk\" (UID: \"b1df5064-2ffa-4312-b6f3-cd6dd02c156a\") " pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:43:54.994998 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.994893    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3f8d9ae-aa9e-4eaa-ae8e-97526bc3add9-cert\") pod \"ingress-canary-6ls6n\" (UID: \"b3f8d9ae-aa9e-4eaa-ae8e-97526bc3add9\") " pod="openshift-ingress-canary/ingress-canary-6ls6n"
Sep 05 19:43:54.994998 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.994933    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xprcm\" (UniqueName: \"kubernetes.io/projected/b3f8d9ae-aa9e-4eaa-ae8e-97526bc3add9-kube-api-access-xprcm\") pod \"ingress-canary-6ls6n\" (UID: \"b3f8d9ae-aa9e-4eaa-ae8e-97526bc3add9\") " pod="openshift-ingress-canary/ingress-canary-6ls6n"
Sep 05 19:43:54.995233 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.995209    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-tmp-dir\") pod \"dns-default-zj8hk\" (UID: \"b1df5064-2ffa-4312-b6f3-cd6dd02c156a\") " pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:43:54.995343 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:54.995329    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-config-volume\") pod \"dns-default-zj8hk\" (UID: \"b1df5064-2ffa-4312-b6f3-cd6dd02c156a\") " pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:43:55.006194 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:55.006139    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p8jd\" (UniqueName: \"kubernetes.io/projected/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-kube-api-access-2p8jd\") pod \"dns-default-zj8hk\" (UID: \"b1df5064-2ffa-4312-b6f3-cd6dd02c156a\") " pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:43:55.096281 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:55.096237    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3f8d9ae-aa9e-4eaa-ae8e-97526bc3add9-cert\") pod \"ingress-canary-6ls6n\" (UID: \"b3f8d9ae-aa9e-4eaa-ae8e-97526bc3add9\") " pod="openshift-ingress-canary/ingress-canary-6ls6n"
Sep 05 19:43:55.096481 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:55.096321    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xprcm\" (UniqueName: \"kubernetes.io/projected/b3f8d9ae-aa9e-4eaa-ae8e-97526bc3add9-kube-api-access-xprcm\") pod \"ingress-canary-6ls6n\" (UID: \"b3f8d9ae-aa9e-4eaa-ae8e-97526bc3add9\") " pod="openshift-ingress-canary/ingress-canary-6ls6n"
Sep 05 19:43:55.099182 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:55.099154    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3f8d9ae-aa9e-4eaa-ae8e-97526bc3add9-cert\") pod \"ingress-canary-6ls6n\" (UID: \"b3f8d9ae-aa9e-4eaa-ae8e-97526bc3add9\") " pod="openshift-ingress-canary/ingress-canary-6ls6n"
Sep 05 19:43:55.106289 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:55.106263    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xprcm\" (UniqueName: \"kubernetes.io/projected/b3f8d9ae-aa9e-4eaa-ae8e-97526bc3add9-kube-api-access-xprcm\") pod \"ingress-canary-6ls6n\" (UID: \"b3f8d9ae-aa9e-4eaa-ae8e-97526bc3add9\") " pod="openshift-ingress-canary/ingress-canary-6ls6n"
Sep 05 19:43:55.145213 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:55.145179    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6ls6n"
Sep 05 19:43:55.428509 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:55.428471    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:43:55.431303 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:55.431283    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 05 19:43:55.431723 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:55.431551    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-srspc\""
Sep 05 19:43:55.499827 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:55.499790    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls\") pod \"dns-default-zj8hk\" (UID: \"b1df5064-2ffa-4312-b6f3-cd6dd02c156a\") " pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:43:55.500016 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:55.499954    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 19:43:55.500074 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:55.500018    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls podName:b1df5064-2ffa-4312-b6f3-cd6dd02c156a nodeName:}" failed. No retries permitted until 2026-09-05 19:43:56.500001559 +0000 UTC m=+35.529633446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls") pod "dns-default-zj8hk" (UID: "b1df5064-2ffa-4312-b6f3-cd6dd02c156a") : secret "dns-default-metrics-tls" not found
Sep 05 19:43:56.054230 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:56.054002    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6ls6n"]
Sep 05 19:43:56.090936 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:43:56.090905    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f8d9ae_aa9e_4eaa_ae8e_97526bc3add9.slice/crio-cc0569c85733d07d9ebf125fa6d3bfec2043be1ffc535a7316650ff1ca6a839f WatchSource:0}: Error finding container cc0569c85733d07d9ebf125fa6d3bfec2043be1ffc535a7316650ff1ca6a839f: Status 404 returned error can't find the container with id cc0569c85733d07d9ebf125fa6d3bfec2043be1ffc535a7316650ff1ca6a839f
Sep 05 19:43:56.428709 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:56.428687    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:43:56.428846 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:56.428686    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:43:56.431739 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:56.431714    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gn74z\""
Sep 05 19:43:56.431739 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:56.431738    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 05 19:43:56.432255 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:56.431760    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 05 19:43:56.432255 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:56.431722    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 05 19:43:56.508057 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:56.508023    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls\") pod \"dns-default-zj8hk\" (UID: \"b1df5064-2ffa-4312-b6f3-cd6dd02c156a\") " pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:43:56.508264 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:56.508155    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 19:43:56.508264 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:56.508218    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls podName:b1df5064-2ffa-4312-b6f3-cd6dd02c156a nodeName:}" failed. No retries permitted until 2026-09-05 19:43:58.508205047 +0000 UTC m=+37.537836929 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls") pod "dns-default-zj8hk" (UID: "b1df5064-2ffa-4312-b6f3-cd6dd02c156a") : secret "dns-default-metrics-tls" not found
Sep 05 19:43:56.755848 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:56.755766    2583 generic.go:356] "Generic (PLEG): container finished" podID="334781cf-bbb8-46db-a50c-4d13f6e9f204" containerID="4f9139f2b20daa9cdc1b0a55d9f292372b9498e7b1e71793057525d1b372d8ef" exitCode=0
Sep 05 19:43:56.755995 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:56.755883    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs5q" event={"ID":"334781cf-bbb8-46db-a50c-4d13f6e9f204","Type":"ContainerDied","Data":"4f9139f2b20daa9cdc1b0a55d9f292372b9498e7b1e71793057525d1b372d8ef"}
Sep 05 19:43:56.757380 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:56.757325    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6ls6n" event={"ID":"b3f8d9ae-aa9e-4eaa-ae8e-97526bc3add9","Type":"ContainerStarted","Data":"cc0569c85733d07d9ebf125fa6d3bfec2043be1ffc535a7316650ff1ca6a839f"}
Sep 05 19:43:57.764188 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:57.764141    2583 generic.go:356] "Generic (PLEG): container finished" podID="334781cf-bbb8-46db-a50c-4d13f6e9f204" containerID="43f8fa9e2a6ba848e5f252da7ae3f7c2b939c4a58c35be4fceb78e5f2e652805" exitCode=0
Sep 05 19:43:57.764829 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:57.764228    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs5q" event={"ID":"334781cf-bbb8-46db-a50c-4d13f6e9f204","Type":"ContainerDied","Data":"43f8fa9e2a6ba848e5f252da7ae3f7c2b939c4a58c35be4fceb78e5f2e652805"}
Sep 05 19:43:58.522972 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:58.522937    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls\") pod \"dns-default-zj8hk\" (UID: \"b1df5064-2ffa-4312-b6f3-cd6dd02c156a\") " pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:43:58.523122 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:58.523052    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 19:43:58.523122 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:43:58.523105    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls podName:b1df5064-2ffa-4312-b6f3-cd6dd02c156a nodeName:}" failed. No retries permitted until 2026-09-05 19:44:02.523089043 +0000 UTC m=+41.552720946 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls") pod "dns-default-zj8hk" (UID: "b1df5064-2ffa-4312-b6f3-cd6dd02c156a") : secret "dns-default-metrics-tls" not found
Sep 05 19:43:58.771013 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:58.770723    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs5q" event={"ID":"334781cf-bbb8-46db-a50c-4d13f6e9f204","Type":"ContainerStarted","Data":"bfccb1d9691e263fc143c0d65e7b23f69818f50c155a92cd7198a42ccec1d777"}
Sep 05 19:43:58.773421 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:58.773393    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6ls6n" event={"ID":"b3f8d9ae-aa9e-4eaa-ae8e-97526bc3add9","Type":"ContainerStarted","Data":"629529b207106b95d1248a641e6f479484bf0861e8457db2ee92a50be173b862"}
Sep 05 19:43:58.796587 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:58.796535    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wzs5q" podStartSLOduration=8.003705927 podStartE2EDuration="37.796519401s" totalImagesPullingTime="29.792813474s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:58.795471974 +0000 UTC m=+37.825103879" watchObservedRunningTime="2026-09-05 19:43:58.796519401 +0000 UTC m=+37.826151305"
Sep 05 19:43:58.817078 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:43:58.817027    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6ls6n" podStartSLOduration=2.294667797 podStartE2EDuration="4.817011925s" totalImagesPullingTime="2.522344128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:43:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:43:58.815995763 +0000 UTC m=+37.845627679" watchObservedRunningTime="2026-09-05 19:43:58.817011925 +0000 UTC m=+37.846643827"
Sep 05 19:44:02.549832 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:02.549784    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls\") pod \"dns-default-zj8hk\" (UID: \"b1df5064-2ffa-4312-b6f3-cd6dd02c156a\") " pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:44:02.550275 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:44:02.549912    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 19:44:02.550275 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:44:02.549965    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls podName:b1df5064-2ffa-4312-b6f3-cd6dd02c156a nodeName:}" failed. No retries permitted until 2026-09-05 19:44:10.54995047 +0000 UTC m=+49.579582352 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls") pod "dns-default-zj8hk" (UID: "b1df5064-2ffa-4312-b6f3-cd6dd02c156a") : secret "dns-default-metrics-tls" not found
Sep 05 19:44:10.607629 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:10.607575    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls\") pod \"dns-default-zj8hk\" (UID: \"b1df5064-2ffa-4312-b6f3-cd6dd02c156a\") " pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:44:10.608011 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:44:10.607718    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 19:44:10.608011 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:44:10.607776    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls podName:b1df5064-2ffa-4312-b6f3-cd6dd02c156a nodeName:}" failed. No retries permitted until 2026-09-05 19:44:26.607761913 +0000 UTC m=+65.637393800 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls") pod "dns-default-zj8hk" (UID: "b1df5064-2ffa-4312-b6f3-cd6dd02c156a") : secret "dns-default-metrics-tls" not found
Sep 05 19:44:21.751794 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:21.751764    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mf9tj"
Sep 05 19:44:26.114735 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:26.114683    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83377d0d-0308-4613-bd53-04d2cb4adc3b-original-pull-secret\") pod \"global-pull-secret-syncer-ccth2\" (UID: \"83377d0d-0308-4613-bd53-04d2cb4adc3b\") " pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:44:26.117514 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:26.117497    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 05 19:44:26.128571 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:26.128548    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/83377d0d-0308-4613-bd53-04d2cb4adc3b-original-pull-secret\") pod \"global-pull-secret-syncer-ccth2\" (UID: \"83377d0d-0308-4613-bd53-04d2cb4adc3b\") " pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:44:26.142735 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:26.142713    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ccth2"
Sep 05 19:44:26.215558 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:26.215526    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs\") pod \"network-metrics-daemon-rwwbl\" (UID: \"b9720241-ac7b-4d79-9644-233caff66a8c\") " pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:44:26.218051 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:26.218032    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 05 19:44:26.225978 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:44:26.225963    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 05 19:44:26.226035 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:44:26.226026    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs podName:b9720241-ac7b-4d79-9644-233caff66a8c nodeName:}" failed. No retries permitted until 2026-09-05 19:45:30.226010157 +0000 UTC m=+129.255642039 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs") pod "network-metrics-daemon-rwwbl" (UID: "b9720241-ac7b-4d79-9644-233caff66a8c") : secret "metrics-daemon-secret" not found
Sep 05 19:44:26.316145 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:26.316102    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckqjv\" (UniqueName: \"kubernetes.io/projected/63646bb5-7fb0-417d-96a8-24d5c8591bf1-kube-api-access-ckqjv\") pod \"network-check-target-zxsbh\" (UID: \"63646bb5-7fb0-417d-96a8-24d5c8591bf1\") " pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:44:26.318889 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:26.318873    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 05 19:44:26.329470 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:26.329453    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 05 19:44:26.340032 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:26.340005    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckqjv\" (UniqueName: \"kubernetes.io/projected/63646bb5-7fb0-417d-96a8-24d5c8591bf1-kube-api-access-ckqjv\") pod \"network-check-target-zxsbh\" (UID: \"63646bb5-7fb0-417d-96a8-24d5c8591bf1\") " pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:44:26.410825 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:26.410760    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ccth2"]
Sep 05 19:44:26.414089 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:44:26.414062    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83377d0d_0308_4613_bd53_04d2cb4adc3b.slice/crio-b4fa19008eb9ee4cf1b2a4d79b520785a4a766b0744d3dca1ef8d5488e958bf3 WatchSource:0}: Error finding container b4fa19008eb9ee4cf1b2a4d79b520785a4a766b0744d3dca1ef8d5488e958bf3: Status 404 returned error can't find the container with id b4fa19008eb9ee4cf1b2a4d79b520785a4a766b0744d3dca1ef8d5488e958bf3
Sep 05 19:44:26.440567 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:26.440542    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gn74z\""
Sep 05 19:44:26.448557 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:26.448541    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:44:26.619382 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:26.619349    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls\") pod \"dns-default-zj8hk\" (UID: \"b1df5064-2ffa-4312-b6f3-cd6dd02c156a\") " pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:44:26.619547 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:44:26.619496    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 19:44:26.619588 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:44:26.619560    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls podName:b1df5064-2ffa-4312-b6f3-cd6dd02c156a nodeName:}" failed. No retries permitted until 2026-09-05 19:44:58.619541304 +0000 UTC m=+97.649173197 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls") pod "dns-default-zj8hk" (UID: "b1df5064-2ffa-4312-b6f3-cd6dd02c156a") : secret "dns-default-metrics-tls" not found
Sep 05 19:44:26.713457 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:26.713388    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zxsbh"]
Sep 05 19:44:26.716510 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:44:26.716488    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63646bb5_7fb0_417d_96a8_24d5c8591bf1.slice/crio-75b0ad5c923d1baa205a55fa184dbc5c7325856cdc032b44c69e64cb4d82dc14 WatchSource:0}: Error finding container 75b0ad5c923d1baa205a55fa184dbc5c7325856cdc032b44c69e64cb4d82dc14: Status 404 returned error can't find the container with id 75b0ad5c923d1baa205a55fa184dbc5c7325856cdc032b44c69e64cb4d82dc14
Sep 05 19:44:26.840996 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:26.840955    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ccth2" event={"ID":"83377d0d-0308-4613-bd53-04d2cb4adc3b","Type":"ContainerStarted","Data":"b4fa19008eb9ee4cf1b2a4d79b520785a4a766b0744d3dca1ef8d5488e958bf3"}
Sep 05 19:44:26.842394 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:26.842364    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zxsbh" event={"ID":"63646bb5-7fb0-417d-96a8-24d5c8591bf1","Type":"ContainerStarted","Data":"75b0ad5c923d1baa205a55fa184dbc5c7325856cdc032b44c69e64cb4d82dc14"}
Sep 05 19:44:31.857774 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:31.857733    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ccth2" event={"ID":"83377d0d-0308-4613-bd53-04d2cb4adc3b","Type":"ContainerStarted","Data":"d1016e90fd0111437e42e97341fee9f195e061564d4582faf98959c034b71ec1"}
Sep 05 19:44:31.859150 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:31.859120    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zxsbh" event={"ID":"63646bb5-7fb0-417d-96a8-24d5c8591bf1","Type":"ContainerStarted","Data":"a981e9bb5cb0d2c2f5b11bb467a3ef6556b6422155c25c8467d018252a71ca6e"}
Sep 05 19:44:31.859274 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:31.859244    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:44:31.877325 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:31.877281    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ccth2" podStartSLOduration=66.103797894 podStartE2EDuration="1m10.877267688s" totalImagesPullingTime="4.773469794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:44:31.876597058 +0000 UTC m=+70.906228963" watchObservedRunningTime="2026-09-05 19:44:31.877267688 +0000 UTC m=+70.906899592"
Sep 05 19:44:31.895130 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:31.895028    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zxsbh" podStartSLOduration=66.425563844 podStartE2EDuration="1m10.895012493s" totalImagesPullingTime="4.469448649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:44:31.89440723 +0000 UTC m=+70.924039135" watchObservedRunningTime="2026-09-05 19:44:31.895012493 +0000 UTC m=+70.924644394"
Sep 05 19:44:58.639171 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:44:58.639102    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls\") pod \"dns-default-zj8hk\" (UID: \"b1df5064-2ffa-4312-b6f3-cd6dd02c156a\") " pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:44:58.639655 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:44:58.639241    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 05 19:44:58.639655 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:44:58.639322    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls podName:b1df5064-2ffa-4312-b6f3-cd6dd02c156a nodeName:}" failed. No retries permitted until 2026-09-05 19:46:02.639303597 +0000 UTC m=+161.668935485 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls") pod "dns-default-zj8hk" (UID: "b1df5064-2ffa-4312-b6f3-cd6dd02c156a") : secret "dns-default-metrics-tls" not found
Sep 05 19:45:02.863470 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:02.863437    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zxsbh"
Sep 05 19:45:30.243148 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:30.243080    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs\") pod \"network-metrics-daemon-rwwbl\" (UID: \"b9720241-ac7b-4d79-9644-233caff66a8c\") " pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:45:30.243643 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:30.243246    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 05 19:45:30.243643 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:30.243297    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs podName:b9720241-ac7b-4d79-9644-233caff66a8c nodeName:}" failed. No retries permitted until 2026-09-05 19:47:32.243283574 +0000 UTC m=+251.272915457 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs") pod "network-metrics-daemon-rwwbl" (UID: "b9720241-ac7b-4d79-9644-233caff66a8c") : secret "metrics-daemon-secret" not found
Sep 05 19:45:35.523057 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.523026    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wm7bd"]
Sep 05 19:45:35.524763 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.524747    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wm7bd"
Sep 05 19:45:35.526715 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.526692    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-r6sg9\""
Sep 05 19:45:35.526886 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.526719    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 05 19:45:35.526886 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.526730    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 05 19:45:35.526886 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.526696    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 05 19:45:35.528070 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.528050    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 05 19:45:35.536367 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.536347    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wm7bd"]
Sep 05 19:45:35.581052 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.581013    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cd029360-95f2-45c6-9648-1fe5fe757822-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-wm7bd\" (UID: \"cd029360-95f2-45c6-9648-1fe5fe757822\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wm7bd"
Sep 05 19:45:35.581219 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.581064    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd029360-95f2-45c6-9648-1fe5fe757822-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wm7bd\" (UID: \"cd029360-95f2-45c6-9648-1fe5fe757822\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wm7bd"
Sep 05 19:45:35.581219 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.581172    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z6rt\" (UniqueName: \"kubernetes.io/projected/cd029360-95f2-45c6-9648-1fe5fe757822-kube-api-access-9z6rt\") pod \"cluster-monitoring-operator-6cdcc96965-wm7bd\" (UID: \"cd029360-95f2-45c6-9648-1fe5fe757822\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wm7bd"
Sep 05 19:45:35.628757 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.628729    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-ktdpg"]
Sep 05 19:45:35.630342 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.630328    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-ktdpg"
Sep 05 19:45:35.632357 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.632342    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-2kgp6\""
Sep 05 19:45:35.632470 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.632443    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 05 19:45:35.632596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.632507    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 05 19:45:35.635858 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.635841    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-54d767fc5d-pv29k"]
Sep 05 19:45:35.637533 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.637520    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:45:35.639594 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.639573    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 05 19:45:35.639716 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.639600    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 05 19:45:35.639716 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.639621    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 05 19:45:35.639869 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.639718    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qc4qq\""
Sep 05 19:45:35.639869 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.639653    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 05 19:45:35.640062 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.640050    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 05 19:45:35.640102 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.640085    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 05 19:45:35.645790 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.645767    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-ktdpg"]
Sep 05 19:45:35.660791 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.660759    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-54d767fc5d-pv29k"]
Sep 05 19:45:35.681540 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.681511    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z6rt\" (UniqueName: \"kubernetes.io/projected/cd029360-95f2-45c6-9648-1fe5fe757822-kube-api-access-9z6rt\") pod \"cluster-monitoring-operator-6cdcc96965-wm7bd\" (UID: \"cd029360-95f2-45c6-9648-1fe5fe757822\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wm7bd"
Sep 05 19:45:35.681749 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.681566    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cd029360-95f2-45c6-9648-1fe5fe757822-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-wm7bd\" (UID: \"cd029360-95f2-45c6-9648-1fe5fe757822\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wm7bd"
Sep 05 19:45:35.681749 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.681725    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd029360-95f2-45c6-9648-1fe5fe757822-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wm7bd\" (UID: \"cd029360-95f2-45c6-9648-1fe5fe757822\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wm7bd"
Sep 05 19:45:35.681866 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:35.681847    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 05 19:45:35.681961 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:35.681951    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd029360-95f2-45c6-9648-1fe5fe757822-cluster-monitoring-operator-tls podName:cd029360-95f2-45c6-9648-1fe5fe757822 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:36.181933003 +0000 UTC m=+135.211564888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cd029360-95f2-45c6-9648-1fe5fe757822-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wm7bd" (UID: "cd029360-95f2-45c6-9648-1fe5fe757822") : secret "cluster-monitoring-operator-tls" not found
Sep 05 19:45:35.682244 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.682226    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cd029360-95f2-45c6-9648-1fe5fe757822-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-wm7bd\" (UID: \"cd029360-95f2-45c6-9648-1fe5fe757822\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wm7bd"
Sep 05 19:45:35.689274 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.689256    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z6rt\" (UniqueName: \"kubernetes.io/projected/cd029360-95f2-45c6-9648-1fe5fe757822-kube-api-access-9z6rt\") pod \"cluster-monitoring-operator-6cdcc96965-wm7bd\" (UID: \"cd029360-95f2-45c6-9648-1fe5fe757822\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wm7bd"
Sep 05 19:45:35.782633 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.782541    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-metrics-certs\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:45:35.782633 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.782609    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn7rf\" (UniqueName: \"kubernetes.io/projected/4876fe64-e8bb-4e38-9076-4aa7eaa057d5-kube-api-access-zn7rf\") pod \"volume-data-source-validator-695c895bd6-ktdpg\" (UID: \"4876fe64-e8bb-4e38-9076-4aa7eaa057d5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-ktdpg"
Sep 05 19:45:35.782807 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.782675    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-stats-auth\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:45:35.782807 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.782737    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-default-certificate\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:45:35.782807 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.782771    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6pxc\" (UniqueName: \"kubernetes.io/projected/a9c02cc0-cabd-4594-b23b-da027765f68f-kube-api-access-r6pxc\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:45:35.782807 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.782788    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9c02cc0-cabd-4594-b23b-da027765f68f-service-ca-bundle\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:45:35.884102 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.884067    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-default-certificate\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:45:35.884229 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.884131    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6pxc\" (UniqueName: \"kubernetes.io/projected/a9c02cc0-cabd-4594-b23b-da027765f68f-kube-api-access-r6pxc\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:45:35.884229 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.884149    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9c02cc0-cabd-4594-b23b-da027765f68f-service-ca-bundle\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:45:35.884229 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.884168    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-metrics-certs\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:45:35.884229 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.884202    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn7rf\" (UniqueName: \"kubernetes.io/projected/4876fe64-e8bb-4e38-9076-4aa7eaa057d5-kube-api-access-zn7rf\") pod \"volume-data-source-validator-695c895bd6-ktdpg\" (UID: \"4876fe64-e8bb-4e38-9076-4aa7eaa057d5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-ktdpg"
Sep 05 19:45:35.884465 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:35.884299    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 05 19:45:35.884465 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:35.884330    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9c02cc0-cabd-4594-b23b-da027765f68f-service-ca-bundle podName:a9c02cc0-cabd-4594-b23b-da027765f68f nodeName:}" failed. No retries permitted until 2026-09-05 19:45:36.384306416 +0000 UTC m=+135.413938320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a9c02cc0-cabd-4594-b23b-da027765f68f-service-ca-bundle") pod "router-default-54d767fc5d-pv29k" (UID: "a9c02cc0-cabd-4594-b23b-da027765f68f") : configmap references non-existent config key: service-ca.crt
Sep 05 19:45:35.884465 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:35.884357    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-metrics-certs podName:a9c02cc0-cabd-4594-b23b-da027765f68f nodeName:}" failed. No retries permitted until 2026-09-05 19:45:36.384346788 +0000 UTC m=+135.413978690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-metrics-certs") pod "router-default-54d767fc5d-pv29k" (UID: "a9c02cc0-cabd-4594-b23b-da027765f68f") : secret "router-metrics-certs-default" not found
Sep 05 19:45:35.884465 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.884376    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-stats-auth\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:45:35.886548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.886526    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-default-certificate\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:45:35.886640 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.886596    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-stats-auth\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:45:35.892229 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.892207    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn7rf\" (UniqueName: \"kubernetes.io/projected/4876fe64-e8bb-4e38-9076-4aa7eaa057d5-kube-api-access-zn7rf\") pod \"volume-data-source-validator-695c895bd6-ktdpg\" (UID: \"4876fe64-e8bb-4e38-9076-4aa7eaa057d5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-ktdpg"
Sep 05 19:45:35.892339 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.892318    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6pxc\" (UniqueName: \"kubernetes.io/projected/a9c02cc0-cabd-4594-b23b-da027765f68f-kube-api-access-r6pxc\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:45:35.939641 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:35.939616    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-ktdpg"
Sep 05 19:45:36.187430 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:36.187395    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd029360-95f2-45c6-9648-1fe5fe757822-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wm7bd\" (UID: \"cd029360-95f2-45c6-9648-1fe5fe757822\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wm7bd"
Sep 05 19:45:36.187622 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:36.187600    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 05 19:45:36.187709 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:36.187698    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd029360-95f2-45c6-9648-1fe5fe757822-cluster-monitoring-operator-tls podName:cd029360-95f2-45c6-9648-1fe5fe757822 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:37.187677133 +0000 UTC m=+136.217309020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cd029360-95f2-45c6-9648-1fe5fe757822-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wm7bd" (UID: "cd029360-95f2-45c6-9648-1fe5fe757822") : secret "cluster-monitoring-operator-tls" not found
Sep 05 19:45:36.202359 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:36.202329    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-ktdpg"]
Sep 05 19:45:36.205461 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:45:36.205434    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4876fe64_e8bb_4e38_9076_4aa7eaa057d5.slice/crio-f8444c88b7b56a10acad39ad81e243ab4f45fe5a4a6b83c026bc1b08b105ea18 WatchSource:0}: Error finding container f8444c88b7b56a10acad39ad81e243ab4f45fe5a4a6b83c026bc1b08b105ea18: Status 404 returned error can't find the container with id f8444c88b7b56a10acad39ad81e243ab4f45fe5a4a6b83c026bc1b08b105ea18
Sep 05 19:45:36.388825 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:36.388790    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9c02cc0-cabd-4594-b23b-da027765f68f-service-ca-bundle\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:45:36.388825 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:36.388829    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-metrics-certs\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:45:36.389017 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:36.388971    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9c02cc0-cabd-4594-b23b-da027765f68f-service-ca-bundle podName:a9c02cc0-cabd-4594-b23b-da027765f68f nodeName:}" failed. No retries permitted until 2026-09-05 19:45:37.388953926 +0000 UTC m=+136.418585809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a9c02cc0-cabd-4594-b23b-da027765f68f-service-ca-bundle") pod "router-default-54d767fc5d-pv29k" (UID: "a9c02cc0-cabd-4594-b23b-da027765f68f") : configmap references non-existent config key: service-ca.crt
Sep 05 19:45:36.389017 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:36.389000    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 05 19:45:36.389093 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:36.389053    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-metrics-certs podName:a9c02cc0-cabd-4594-b23b-da027765f68f nodeName:}" failed. No retries permitted until 2026-09-05 19:45:37.389041449 +0000 UTC m=+136.418673331 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-metrics-certs") pod "router-default-54d767fc5d-pv29k" (UID: "a9c02cc0-cabd-4594-b23b-da027765f68f") : secret "router-metrics-certs-default" not found
Sep 05 19:45:36.531089 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:36.531009    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-4v2k6"]
Sep 05 19:45:36.533097 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:36.533080    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-4v2k6"
Sep 05 19:45:36.535125 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:36.535080    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 05 19:45:36.535125 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:36.535092    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gnpz4\""
Sep 05 19:45:36.535301 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:36.535132    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 05 19:45:36.535301 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:36.535132    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 05 19:45:36.535301 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:36.535134    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 05 19:45:36.547778 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:36.547760    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-4v2k6"]
Sep 05 19:45:36.691347 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:36.691310    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-4v2k6\" (UID: \"5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-4v2k6"
Sep 05 19:45:36.691502 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:36.691466    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmgtj\" (UniqueName: \"kubernetes.io/projected/5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88-kube-api-access-vmgtj\") pod \"kube-storage-version-migrator-operator-8d7dd4675-4v2k6\" (UID: \"5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-4v2k6"
Sep 05 19:45:36.691577 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:36.691522    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-4v2k6\" (UID: \"5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-4v2k6"
Sep 05 19:45:36.792916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:36.792830    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-4v2k6\" (UID: \"5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-4v2k6"
Sep 05 19:45:36.793089 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:36.792938    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-4v2k6\" (UID: \"5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-4v2k6"
Sep 05 19:45:36.793089 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:36.792988    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmgtj\" (UniqueName: \"kubernetes.io/projected/5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88-kube-api-access-vmgtj\") pod \"kube-storage-version-migrator-operator-8d7dd4675-4v2k6\" (UID: \"5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-4v2k6"
Sep 05 19:45:36.793676 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:36.793647    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-4v2k6\" (UID: \"5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-4v2k6"
Sep 05 19:45:36.795657 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:36.795630    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-4v2k6\" (UID: \"5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-4v2k6"
Sep 05 19:45:36.802016 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:36.801992    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmgtj\" (UniqueName: \"kubernetes.io/projected/5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88-kube-api-access-vmgtj\") pod \"kube-storage-version-migrator-operator-8d7dd4675-4v2k6\" (UID: \"5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-4v2k6"
Sep 05 19:45:36.841934 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:36.841901    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-4v2k6"
Sep 05 19:45:37.015859 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:37.015821    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-ktdpg" event={"ID":"4876fe64-e8bb-4e38-9076-4aa7eaa057d5","Type":"ContainerStarted","Data":"f8444c88b7b56a10acad39ad81e243ab4f45fe5a4a6b83c026bc1b08b105ea18"}
Sep 05 19:45:37.117366 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:37.117330    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-4v2k6"]
Sep 05 19:45:37.195538 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:37.195500    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd029360-95f2-45c6-9648-1fe5fe757822-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wm7bd\" (UID: \"cd029360-95f2-45c6-9648-1fe5fe757822\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wm7bd"
Sep 05 19:45:37.195718 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:37.195660    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 05 19:45:37.195771 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:37.195756    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd029360-95f2-45c6-9648-1fe5fe757822-cluster-monitoring-operator-tls podName:cd029360-95f2-45c6-9648-1fe5fe757822 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:39.195733848 +0000 UTC m=+138.225365733 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cd029360-95f2-45c6-9648-1fe5fe757822-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wm7bd" (UID: "cd029360-95f2-45c6-9648-1fe5fe757822") : secret "cluster-monitoring-operator-tls" not found
Sep 05 19:45:37.232146 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:45:37.232101    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2c0c10_9d8c_4ceb_adb3_a963ef9a9a88.slice/crio-c73df0cea9dbef75c8b4cd469735cfbb3debc6bec1f6ee0d3140a7836188c068 WatchSource:0}: Error finding container c73df0cea9dbef75c8b4cd469735cfbb3debc6bec1f6ee0d3140a7836188c068: Status 404 returned error can't find the container with id c73df0cea9dbef75c8b4cd469735cfbb3debc6bec1f6ee0d3140a7836188c068
Sep 05 19:45:37.396726 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:37.396642    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9c02cc0-cabd-4594-b23b-da027765f68f-service-ca-bundle\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:45:37.396726 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:37.396683    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-metrics-certs\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:45:37.396898 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:37.396783    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 05 19:45:37.396898 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:37.396808    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9c02cc0-cabd-4594-b23b-da027765f68f-service-ca-bundle podName:a9c02cc0-cabd-4594-b23b-da027765f68f nodeName:}" failed. No retries permitted until 2026-09-05 19:45:39.396791374 +0000 UTC m=+138.426423256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a9c02cc0-cabd-4594-b23b-da027765f68f-service-ca-bundle") pod "router-default-54d767fc5d-pv29k" (UID: "a9c02cc0-cabd-4594-b23b-da027765f68f") : configmap references non-existent config key: service-ca.crt
Sep 05 19:45:37.396898 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:37.396874    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-metrics-certs podName:a9c02cc0-cabd-4594-b23b-da027765f68f nodeName:}" failed. No retries permitted until 2026-09-05 19:45:39.396866209 +0000 UTC m=+138.426498091 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-metrics-certs") pod "router-default-54d767fc5d-pv29k" (UID: "a9c02cc0-cabd-4594-b23b-da027765f68f") : secret "router-metrics-certs-default" not found
Sep 05 19:45:38.020034 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:38.020000    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-ktdpg" event={"ID":"4876fe64-e8bb-4e38-9076-4aa7eaa057d5","Type":"ContainerStarted","Data":"ac45031e733f535a4b891a9cf669e32671aa46bfdb1b31c4e53a73b914a864a6"}
Sep 05 19:45:38.021150 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:38.021126    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-4v2k6" event={"ID":"5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88","Type":"ContainerStarted","Data":"c73df0cea9dbef75c8b4cd469735cfbb3debc6bec1f6ee0d3140a7836188c068"}
Sep 05 19:45:38.034815 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:38.034772    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-ktdpg" podStartSLOduration=1.990353477 podStartE2EDuration="3.034756442s" totalImagesPullingTime="1.044402965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:45:38.033961537 +0000 UTC m=+137.063593442" watchObservedRunningTime="2026-09-05 19:45:38.034756442 +0000 UTC m=+137.064388347"
Sep 05 19:45:39.210504 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:39.210450    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd029360-95f2-45c6-9648-1fe5fe757822-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wm7bd\" (UID: \"cd029360-95f2-45c6-9648-1fe5fe757822\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wm7bd"
Sep 05 19:45:39.210860 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:39.210608    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 05 19:45:39.210860 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:39.210679    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd029360-95f2-45c6-9648-1fe5fe757822-cluster-monitoring-operator-tls podName:cd029360-95f2-45c6-9648-1fe5fe757822 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:43.21066043 +0000 UTC m=+142.240292335 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cd029360-95f2-45c6-9648-1fe5fe757822-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wm7bd" (UID: "cd029360-95f2-45c6-9648-1fe5fe757822") : secret "cluster-monitoring-operator-tls" not found
Sep 05 19:45:39.412116 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:39.412073    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9c02cc0-cabd-4594-b23b-da027765f68f-service-ca-bundle\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:45:39.412271 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:39.412129    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-metrics-certs\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:45:39.412271 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:39.412238    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 05 19:45:39.412271 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:39.412245    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9c02cc0-cabd-4594-b23b-da027765f68f-service-ca-bundle podName:a9c02cc0-cabd-4594-b23b-da027765f68f nodeName:}" failed. No retries permitted until 2026-09-05 19:45:43.412227825 +0000 UTC m=+142.441859712 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a9c02cc0-cabd-4594-b23b-da027765f68f-service-ca-bundle") pod "router-default-54d767fc5d-pv29k" (UID: "a9c02cc0-cabd-4594-b23b-da027765f68f") : configmap references non-existent config key: service-ca.crt
Sep 05 19:45:39.412386 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:39.412276    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-metrics-certs podName:a9c02cc0-cabd-4594-b23b-da027765f68f nodeName:}" failed. No retries permitted until 2026-09-05 19:45:43.412265033 +0000 UTC m=+142.441896915 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-metrics-certs") pod "router-default-54d767fc5d-pv29k" (UID: "a9c02cc0-cabd-4594-b23b-da027765f68f") : secret "router-metrics-certs-default" not found
Sep 05 19:45:39.576718 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:39.576684    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pgxbk"]
Sep 05 19:45:39.578514 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:39.578499    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:39.580545 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:39.580521    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 05 19:45:39.580663 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:39.580528    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 05 19:45:39.581005 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:39.580986    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 05 19:45:39.581005 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:39.580992    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 05 19:45:39.581210 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:39.581011    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wszhj\""
Sep 05 19:45:39.590817 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:39.590798    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pgxbk"]
Sep 05 19:45:39.714974 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:39.714890    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:39.714974 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:39.714924    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-crio-socket\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:39.714974 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:39.714963    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:39.715263 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:39.715060    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-data-volume\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:39.715263 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:39.715166    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npg2g\" (UniqueName: \"kubernetes.io/projected/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-kube-api-access-npg2g\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:39.819984 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:39.816703    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-data-volume\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:39.819984 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:39.816827    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npg2g\" (UniqueName: \"kubernetes.io/projected/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-kube-api-access-npg2g\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:39.819984 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:39.816876    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:39.819984 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:39.816912    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-crio-socket\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:39.819984 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:39.816970    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:39.819984 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:39.817134    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 19:45:39.819984 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:39.817205    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-insights-runtime-extractor-tls podName:7a9b2788-2ab7-4c18-b80d-b5192c2909e0 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:40.31718612 +0000 UTC m=+139.346818018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgxbk" (UID: "7a9b2788-2ab7-4c18-b80d-b5192c2909e0") : secret "insights-runtime-extractor-tls" not found
Sep 05 19:45:39.819984 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:39.817801    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-data-volume\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:39.819984 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:39.818171    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 19:45:39.819984 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:39.818229    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-kube-rbac-proxy-cm podName:7a9b2788-2ab7-4c18-b80d-b5192c2909e0 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:40.318212788 +0000 UTC m=+139.347844685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pgxbk" (UID: "7a9b2788-2ab7-4c18-b80d-b5192c2909e0") : configmap "kube-rbac-proxy" not found
Sep 05 19:45:39.819984 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:39.818292    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-crio-socket\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:39.829230 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:39.829204    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npg2g\" (UniqueName: \"kubernetes.io/projected/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-kube-api-access-npg2g\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:40.028696 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:40.028593    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-4v2k6" event={"ID":"5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88","Type":"ContainerStarted","Data":"02bc1e6089e6ec38353873e9525d3f9d63d8d5c79ecfe41744b4a757fd9865c7"}
Sep 05 19:45:40.044579 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:40.044528    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-4v2k6" podStartSLOduration=2.243781434 podStartE2EDuration="4.044514484s" totalImagesPullingTime="1.80073305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:45:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:45:40.043914407 +0000 UTC m=+139.073546313" watchObservedRunningTime="2026-09-05 19:45:40.044514484 +0000 UTC m=+139.074146402"
Sep 05 19:45:40.321483 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:40.321432    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:40.321858 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:40.321501    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:40.321858 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:40.321552    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 19:45:40.321858 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:40.321632    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-kube-rbac-proxy-cm podName:7a9b2788-2ab7-4c18-b80d-b5192c2909e0 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:41.321612752 +0000 UTC m=+140.351244664 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pgxbk" (UID: "7a9b2788-2ab7-4c18-b80d-b5192c2909e0") : configmap "kube-rbac-proxy" not found
Sep 05 19:45:40.321858 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:40.321638    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 19:45:40.321858 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:40.321677    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-insights-runtime-extractor-tls podName:7a9b2788-2ab7-4c18-b80d-b5192c2909e0 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:41.321667102 +0000 UTC m=+140.351299000 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgxbk" (UID: "7a9b2788-2ab7-4c18-b80d-b5192c2909e0") : secret "insights-runtime-extractor-tls" not found
Sep 05 19:45:41.330614 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:41.330574    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:41.330980 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:41.330631    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:41.330980 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:41.330690    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 19:45:41.330980 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:41.330739    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 19:45:41.330980 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:41.330750    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-kube-rbac-proxy-cm podName:7a9b2788-2ab7-4c18-b80d-b5192c2909e0 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:43.330736219 +0000 UTC m=+142.360368105 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pgxbk" (UID: "7a9b2788-2ab7-4c18-b80d-b5192c2909e0") : configmap "kube-rbac-proxy" not found
Sep 05 19:45:41.330980 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:41.330778    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-insights-runtime-extractor-tls podName:7a9b2788-2ab7-4c18-b80d-b5192c2909e0 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:43.3307665 +0000 UTC m=+142.360398383 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgxbk" (UID: "7a9b2788-2ab7-4c18-b80d-b5192c2909e0") : secret "insights-runtime-extractor-tls" not found
Sep 05 19:45:41.537639 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:41.537587    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5h45d_0216ba9d-7122-47a3-86eb-8d14ddb478b0/dns-node-resolver/0.log"
Sep 05 19:45:42.740668 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:42.740623    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z6kd9_02b1f61a-c064-4c08-9bd4-cbe7fe25c934/node-ca/0.log"
Sep 05 19:45:43.136798 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:43.136767    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6ls6n_b3f8d9ae-aa9e-4eaa-ae8e-97526bc3add9/serve-healthcheck-canary/0.log"
Sep 05 19:45:43.244681 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:43.244644    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd029360-95f2-45c6-9648-1fe5fe757822-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wm7bd\" (UID: \"cd029360-95f2-45c6-9648-1fe5fe757822\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wm7bd"
Sep 05 19:45:43.244791 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:43.244757    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 05 19:45:43.244850 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:43.244813    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd029360-95f2-45c6-9648-1fe5fe757822-cluster-monitoring-operator-tls podName:cd029360-95f2-45c6-9648-1fe5fe757822 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:51.244797892 +0000 UTC m=+150.274429775 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cd029360-95f2-45c6-9648-1fe5fe757822-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wm7bd" (UID: "cd029360-95f2-45c6-9648-1fe5fe757822") : secret "cluster-monitoring-operator-tls" not found
Sep 05 19:45:43.345837 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:43.345807    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:43.345949 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:43.345860    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:43.346013 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:43.345991    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 19:45:43.346065 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:43.346055    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-kube-rbac-proxy-cm podName:7a9b2788-2ab7-4c18-b80d-b5192c2909e0 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:47.346040625 +0000 UTC m=+146.375672508 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pgxbk" (UID: "7a9b2788-2ab7-4c18-b80d-b5192c2909e0") : configmap "kube-rbac-proxy" not found
Sep 05 19:45:43.346142 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:43.345988    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 19:45:43.346197 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:43.346165    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-insights-runtime-extractor-tls podName:7a9b2788-2ab7-4c18-b80d-b5192c2909e0 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:47.346147715 +0000 UTC m=+146.375779608 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgxbk" (UID: "7a9b2788-2ab7-4c18-b80d-b5192c2909e0") : secret "insights-runtime-extractor-tls" not found
Sep 05 19:45:43.446381 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:43.446310    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9c02cc0-cabd-4594-b23b-da027765f68f-service-ca-bundle\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:45:43.446381 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:43.446345    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-metrics-certs\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:45:43.446530 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:43.446450    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 05 19:45:43.446530 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:43.446485    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9c02cc0-cabd-4594-b23b-da027765f68f-service-ca-bundle podName:a9c02cc0-cabd-4594-b23b-da027765f68f nodeName:}" failed. No retries permitted until 2026-09-05 19:45:51.446466861 +0000 UTC m=+150.476098767 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a9c02cc0-cabd-4594-b23b-da027765f68f-service-ca-bundle") pod "router-default-54d767fc5d-pv29k" (UID: "a9c02cc0-cabd-4594-b23b-da027765f68f") : configmap references non-existent config key: service-ca.crt
Sep 05 19:45:43.446530 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:43.446503    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-metrics-certs podName:a9c02cc0-cabd-4594-b23b-da027765f68f nodeName:}" failed. No retries permitted until 2026-09-05 19:45:51.446496591 +0000 UTC m=+150.476128476 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-metrics-certs") pod "router-default-54d767fc5d-pv29k" (UID: "a9c02cc0-cabd-4594-b23b-da027765f68f") : secret "router-metrics-certs-default" not found
Sep 05 19:45:47.376087 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:47.376048    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:47.376683 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:47.376101    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:47.376683 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:47.376219    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 05 19:45:47.376683 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:47.376307    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-kube-rbac-proxy-cm podName:7a9b2788-2ab7-4c18-b80d-b5192c2909e0 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:55.376284692 +0000 UTC m=+154.405916592 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pgxbk" (UID: "7a9b2788-2ab7-4c18-b80d-b5192c2909e0") : configmap "kube-rbac-proxy" not found
Sep 05 19:45:47.376683 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:47.376225    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 19:45:47.376683 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:47.376351    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-insights-runtime-extractor-tls podName:7a9b2788-2ab7-4c18-b80d-b5192c2909e0 nodeName:}" failed. No retries permitted until 2026-09-05 19:45:55.376340549 +0000 UTC m=+154.405972445 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgxbk" (UID: "7a9b2788-2ab7-4c18-b80d-b5192c2909e0") : secret "insights-runtime-extractor-tls" not found
Sep 05 19:45:51.304285 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:51.304246    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd029360-95f2-45c6-9648-1fe5fe757822-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wm7bd\" (UID: \"cd029360-95f2-45c6-9648-1fe5fe757822\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wm7bd"
Sep 05 19:45:51.304658 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:51.304390    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 05 19:45:51.304658 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:51.304450    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd029360-95f2-45c6-9648-1fe5fe757822-cluster-monitoring-operator-tls podName:cd029360-95f2-45c6-9648-1fe5fe757822 nodeName:}" failed. No retries permitted until 2026-09-05 19:46:07.304435902 +0000 UTC m=+166.334067784 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cd029360-95f2-45c6-9648-1fe5fe757822-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-wm7bd" (UID: "cd029360-95f2-45c6-9648-1fe5fe757822") : secret "cluster-monitoring-operator-tls" not found
Sep 05 19:45:51.506241 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:51.506211    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9c02cc0-cabd-4594-b23b-da027765f68f-service-ca-bundle\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:45:51.506241 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:51.506245    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-metrics-certs\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:45:51.506474 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:51.506331    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 05 19:45:51.506474 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:51.506380    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-metrics-certs podName:a9c02cc0-cabd-4594-b23b-da027765f68f nodeName:}" failed. No retries permitted until 2026-09-05 19:46:07.506367219 +0000 UTC m=+166.535999102 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-metrics-certs") pod "router-default-54d767fc5d-pv29k" (UID: "a9c02cc0-cabd-4594-b23b-da027765f68f") : secret "router-metrics-certs-default" not found
Sep 05 19:45:51.506474 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:51.506392    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9c02cc0-cabd-4594-b23b-da027765f68f-service-ca-bundle podName:a9c02cc0-cabd-4594-b23b-da027765f68f nodeName:}" failed. No retries permitted until 2026-09-05 19:46:07.506387129 +0000 UTC m=+166.536019012 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a9c02cc0-cabd-4594-b23b-da027765f68f-service-ca-bundle") pod "router-default-54d767fc5d-pv29k" (UID: "a9c02cc0-cabd-4594-b23b-da027765f68f") : configmap references non-existent config key: service-ca.crt
Sep 05 19:45:55.436803 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:55.436757    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:55.437220 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:55.436853    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:55.437220 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:55.436902    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 05 19:45:55.437220 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:55.436966    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-insights-runtime-extractor-tls podName:7a9b2788-2ab7-4c18-b80d-b5192c2909e0 nodeName:}" failed. No retries permitted until 2026-09-05 19:46:11.436948662 +0000 UTC m=+170.466580547 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pgxbk" (UID: "7a9b2788-2ab7-4c18-b80d-b5192c2909e0") : secret "insights-runtime-extractor-tls" not found
Sep 05 19:45:55.437348 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:55.437302    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:45:57.833740 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:57.833684    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zj8hk" podUID="b1df5064-2ffa-4312-b6f3-cd6dd02c156a"
Sep 05 19:45:58.078966 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:45:58.078929    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:45:58.438928 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:45:58.438893    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-rwwbl" podUID="b9720241-ac7b-4d79-9644-233caff66a8c"
Sep 05 19:46:02.688006 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:02.687961    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls\") pod \"dns-default-zj8hk\" (UID: \"b1df5064-2ffa-4312-b6f3-cd6dd02c156a\") " pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:46:02.690306 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:02.690285    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1df5064-2ffa-4312-b6f3-cd6dd02c156a-metrics-tls\") pod \"dns-default-zj8hk\" (UID: \"b1df5064-2ffa-4312-b6f3-cd6dd02c156a\") " pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:46:02.881339 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:02.881306    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6x9hn\""
Sep 05 19:46:02.889813 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:02.889791    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:46:03.161554 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:03.161522    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zj8hk"]
Sep 05 19:46:03.164677 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:46:03.164649    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1df5064_2ffa_4312_b6f3_cd6dd02c156a.slice/crio-4f09b468876c1bd7f2ee4f61d27ac70a93b1e4f409ea3d69dd93b67099ed0d9c WatchSource:0}: Error finding container 4f09b468876c1bd7f2ee4f61d27ac70a93b1e4f409ea3d69dd93b67099ed0d9c: Status 404 returned error can't find the container with id 4f09b468876c1bd7f2ee4f61d27ac70a93b1e4f409ea3d69dd93b67099ed0d9c
Sep 05 19:46:04.097274 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:04.097235    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zj8hk" event={"ID":"b1df5064-2ffa-4312-b6f3-cd6dd02c156a","Type":"ContainerStarted","Data":"4f09b468876c1bd7f2ee4f61d27ac70a93b1e4f409ea3d69dd93b67099ed0d9c"}
Sep 05 19:46:04.724654 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:04.724623    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fd756675f-mnwfn"]
Sep 05 19:46:04.728781 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:04.728761    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:04.730801 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:04.730749    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 05 19:46:04.730942 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:04.730863    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 05 19:46:04.730988 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:04.730963    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 05 19:46:04.731104 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:04.731087    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rgzgk\""
Sep 05 19:46:04.737197 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:04.737174    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 05 19:46:04.742430 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:04.742408    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fd756675f-mnwfn"]
Sep 05 19:46:04.903701 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:04.903657    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c26eb860-b9c1-40d3-a112-44e5fca931b0-installation-pull-secrets\") pod \"image-registry-fd756675f-mnwfn\" (UID: \"c26eb860-b9c1-40d3-a112-44e5fca931b0\") " pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:04.903701 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:04.903703    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c26eb860-b9c1-40d3-a112-44e5fca931b0-bound-sa-token\") pod \"image-registry-fd756675f-mnwfn\" (UID: \"c26eb860-b9c1-40d3-a112-44e5fca931b0\") " pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:04.903933 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:04.903726    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c26eb860-b9c1-40d3-a112-44e5fca931b0-trusted-ca\") pod \"image-registry-fd756675f-mnwfn\" (UID: \"c26eb860-b9c1-40d3-a112-44e5fca931b0\") " pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:04.903933 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:04.903783    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c26eb860-b9c1-40d3-a112-44e5fca931b0-registry-tls\") pod \"image-registry-fd756675f-mnwfn\" (UID: \"c26eb860-b9c1-40d3-a112-44e5fca931b0\") " pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:04.903933 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:04.903810    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c26eb860-b9c1-40d3-a112-44e5fca931b0-registry-certificates\") pod \"image-registry-fd756675f-mnwfn\" (UID: \"c26eb860-b9c1-40d3-a112-44e5fca931b0\") " pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:04.903933 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:04.903836    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m6b2\" (UniqueName: \"kubernetes.io/projected/c26eb860-b9c1-40d3-a112-44e5fca931b0-kube-api-access-7m6b2\") pod \"image-registry-fd756675f-mnwfn\" (UID: \"c26eb860-b9c1-40d3-a112-44e5fca931b0\") " pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:04.903933 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:04.903910    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c26eb860-b9c1-40d3-a112-44e5fca931b0-image-registry-private-configuration\") pod \"image-registry-fd756675f-mnwfn\" (UID: \"c26eb860-b9c1-40d3-a112-44e5fca931b0\") " pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:04.904136 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:04.903956    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c26eb860-b9c1-40d3-a112-44e5fca931b0-ca-trust-extracted\") pod \"image-registry-fd756675f-mnwfn\" (UID: \"c26eb860-b9c1-40d3-a112-44e5fca931b0\") " pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:05.004859 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:05.004780    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c26eb860-b9c1-40d3-a112-44e5fca931b0-installation-pull-secrets\") pod \"image-registry-fd756675f-mnwfn\" (UID: \"c26eb860-b9c1-40d3-a112-44e5fca931b0\") " pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:05.004859 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:05.004814    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c26eb860-b9c1-40d3-a112-44e5fca931b0-bound-sa-token\") pod \"image-registry-fd756675f-mnwfn\" (UID: \"c26eb860-b9c1-40d3-a112-44e5fca931b0\") " pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:05.004859 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:05.004830    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c26eb860-b9c1-40d3-a112-44e5fca931b0-trusted-ca\") pod \"image-registry-fd756675f-mnwfn\" (UID: \"c26eb860-b9c1-40d3-a112-44e5fca931b0\") " pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:05.005101 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:05.004873    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c26eb860-b9c1-40d3-a112-44e5fca931b0-registry-tls\") pod \"image-registry-fd756675f-mnwfn\" (UID: \"c26eb860-b9c1-40d3-a112-44e5fca931b0\") " pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:05.005101 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:05.004902    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c26eb860-b9c1-40d3-a112-44e5fca931b0-registry-certificates\") pod \"image-registry-fd756675f-mnwfn\" (UID: \"c26eb860-b9c1-40d3-a112-44e5fca931b0\") " pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:05.005101 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:05.004931    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m6b2\" (UniqueName: \"kubernetes.io/projected/c26eb860-b9c1-40d3-a112-44e5fca931b0-kube-api-access-7m6b2\") pod \"image-registry-fd756675f-mnwfn\" (UID: \"c26eb860-b9c1-40d3-a112-44e5fca931b0\") " pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:05.005101 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:05.005006    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c26eb860-b9c1-40d3-a112-44e5fca931b0-image-registry-private-configuration\") pod \"image-registry-fd756675f-mnwfn\" (UID: \"c26eb860-b9c1-40d3-a112-44e5fca931b0\") " pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:05.005101 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:05.005033    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c26eb860-b9c1-40d3-a112-44e5fca931b0-ca-trust-extracted\") pod \"image-registry-fd756675f-mnwfn\" (UID: \"c26eb860-b9c1-40d3-a112-44e5fca931b0\") " pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:05.005791 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:05.005759    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c26eb860-b9c1-40d3-a112-44e5fca931b0-ca-trust-extracted\") pod \"image-registry-fd756675f-mnwfn\" (UID: \"c26eb860-b9c1-40d3-a112-44e5fca931b0\") " pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:05.005911 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:05.005848    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c26eb860-b9c1-40d3-a112-44e5fca931b0-registry-certificates\") pod \"image-registry-fd756675f-mnwfn\" (UID: \"c26eb860-b9c1-40d3-a112-44e5fca931b0\") " pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:05.005911 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:05.005866    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c26eb860-b9c1-40d3-a112-44e5fca931b0-trusted-ca\") pod \"image-registry-fd756675f-mnwfn\" (UID: \"c26eb860-b9c1-40d3-a112-44e5fca931b0\") " pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:05.007559 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:05.007532    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c26eb860-b9c1-40d3-a112-44e5fca931b0-registry-tls\") pod \"image-registry-fd756675f-mnwfn\" (UID: \"c26eb860-b9c1-40d3-a112-44e5fca931b0\") " pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:05.007700 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:05.007683    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c26eb860-b9c1-40d3-a112-44e5fca931b0-image-registry-private-configuration\") pod \"image-registry-fd756675f-mnwfn\" (UID: \"c26eb860-b9c1-40d3-a112-44e5fca931b0\") " pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:05.007769 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:05.007751    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c26eb860-b9c1-40d3-a112-44e5fca931b0-installation-pull-secrets\") pod \"image-registry-fd756675f-mnwfn\" (UID: \"c26eb860-b9c1-40d3-a112-44e5fca931b0\") " pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:05.012466 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:05.012440    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m6b2\" (UniqueName: \"kubernetes.io/projected/c26eb860-b9c1-40d3-a112-44e5fca931b0-kube-api-access-7m6b2\") pod \"image-registry-fd756675f-mnwfn\" (UID: \"c26eb860-b9c1-40d3-a112-44e5fca931b0\") " pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:05.012773 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:05.012749    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c26eb860-b9c1-40d3-a112-44e5fca931b0-bound-sa-token\") pod \"image-registry-fd756675f-mnwfn\" (UID: \"c26eb860-b9c1-40d3-a112-44e5fca931b0\") " pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:05.037952 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:05.037924    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:05.102031 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:05.102006    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zj8hk" event={"ID":"b1df5064-2ffa-4312-b6f3-cd6dd02c156a","Type":"ContainerStarted","Data":"e4cd52ddfb1a170dcdade958ea1cc08ddebed32e7b0f8fff5505a6aba699c3e9"}
Sep 05 19:46:05.102364 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:05.102054    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zj8hk" event={"ID":"b1df5064-2ffa-4312-b6f3-cd6dd02c156a","Type":"ContainerStarted","Data":"b6203a869394c2262aea87ab47443c94a3b2b70ff86d3c6e66300ddbc58fb8a1"}
Sep 05 19:46:05.102364 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:05.102169    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:46:05.118954 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:05.118911    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zj8hk" podStartSLOduration=129.62880743 podStartE2EDuration="2m11.118896162s" totalImagesPullingTime="1.490088732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:43:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:46:05.11762728 +0000 UTC m=+164.147259185" watchObservedRunningTime="2026-09-05 19:46:05.118896162 +0000 UTC m=+164.148528068"
Sep 05 19:46:05.307951 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:05.307880    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fd756675f-mnwfn"]
Sep 05 19:46:05.310858 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:46:05.310833    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26eb860_b9c1_40d3_a112_44e5fca931b0.slice/crio-e33ce4cd34b4218ae54c8220b57a070d5f9f25334ef56f7822585ad7189bd64c WatchSource:0}: Error finding container e33ce4cd34b4218ae54c8220b57a070d5f9f25334ef56f7822585ad7189bd64c: Status 404 returned error can't find the container with id e33ce4cd34b4218ae54c8220b57a070d5f9f25334ef56f7822585ad7189bd64c
Sep 05 19:46:06.106364 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:06.106331    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fd756675f-mnwfn" event={"ID":"c26eb860-b9c1-40d3-a112-44e5fca931b0","Type":"ContainerStarted","Data":"65c70737e0e094f5f1a18486542741e479bde0d6bb2b0d682e17ef88e80c5d9f"}
Sep 05 19:46:06.106757 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:06.106389    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fd756675f-mnwfn" event={"ID":"c26eb860-b9c1-40d3-a112-44e5fca931b0","Type":"ContainerStarted","Data":"e33ce4cd34b4218ae54c8220b57a070d5f9f25334ef56f7822585ad7189bd64c"}
Sep 05 19:46:06.106757 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:06.106484    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:06.136624 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:06.136580    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-fd756675f-mnwfn" podStartSLOduration=2.136565659 podStartE2EDuration="2.136565659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:46:06.135827143 +0000 UTC m=+165.165459049" watchObservedRunningTime="2026-09-05 19:46:06.136565659 +0000 UTC m=+165.166197564"
Sep 05 19:46:07.324210 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:07.324159    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd029360-95f2-45c6-9648-1fe5fe757822-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wm7bd\" (UID: \"cd029360-95f2-45c6-9648-1fe5fe757822\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wm7bd"
Sep 05 19:46:07.326548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:07.326528    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd029360-95f2-45c6-9648-1fe5fe757822-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-wm7bd\" (UID: \"cd029360-95f2-45c6-9648-1fe5fe757822\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wm7bd"
Sep 05 19:46:07.333050 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:07.333030    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wm7bd"
Sep 05 19:46:07.526737 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:07.526701    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9c02cc0-cabd-4594-b23b-da027765f68f-service-ca-bundle\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:46:07.526900 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:07.526744    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-metrics-certs\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:46:07.527361 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:07.527342    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9c02cc0-cabd-4594-b23b-da027765f68f-service-ca-bundle\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:46:07.528972 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:07.528954    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9c02cc0-cabd-4594-b23b-da027765f68f-metrics-certs\") pod \"router-default-54d767fc5d-pv29k\" (UID: \"a9c02cc0-cabd-4594-b23b-da027765f68f\") " pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:46:07.602864 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:07.602716    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wm7bd"]
Sep 05 19:46:07.605123 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:46:07.605072    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd029360_95f2_45c6_9648_1fe5fe757822.slice/crio-e7a9a5ef80c43f4b653cdaa0f9706dddd5f5beffad06ecba50d005c1e7c5ba97 WatchSource:0}: Error finding container e7a9a5ef80c43f4b653cdaa0f9706dddd5f5beffad06ecba50d005c1e7c5ba97: Status 404 returned error can't find the container with id e7a9a5ef80c43f4b653cdaa0f9706dddd5f5beffad06ecba50d005c1e7c5ba97
Sep 05 19:46:07.747225 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:07.747196    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:46:08.021614 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:08.021540    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-54d767fc5d-pv29k"]
Sep 05 19:46:08.024147 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:46:08.024119    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c02cc0_cabd_4594_b23b_da027765f68f.slice/crio-1b62a736b9e4d3d31a14543c48c9cccb7cf0406b8c983e401b46ec1851e043fc WatchSource:0}: Error finding container 1b62a736b9e4d3d31a14543c48c9cccb7cf0406b8c983e401b46ec1851e043fc: Status 404 returned error can't find the container with id 1b62a736b9e4d3d31a14543c48c9cccb7cf0406b8c983e401b46ec1851e043fc
Sep 05 19:46:08.113589 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:08.113552    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wm7bd" event={"ID":"cd029360-95f2-45c6-9648-1fe5fe757822","Type":"ContainerStarted","Data":"e7a9a5ef80c43f4b653cdaa0f9706dddd5f5beffad06ecba50d005c1e7c5ba97"}
Sep 05 19:46:08.115068 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:08.115040    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-54d767fc5d-pv29k" event={"ID":"a9c02cc0-cabd-4594-b23b-da027765f68f","Type":"ContainerStarted","Data":"b4f926b184f6120ccca34c0607757ed3647c3400fc115e4cff92959428b2a384"}
Sep 05 19:46:08.115222 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:08.115089    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-54d767fc5d-pv29k" event={"ID":"a9c02cc0-cabd-4594-b23b-da027765f68f","Type":"ContainerStarted","Data":"1b62a736b9e4d3d31a14543c48c9cccb7cf0406b8c983e401b46ec1851e043fc"}
Sep 05 19:46:08.133409 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:08.133360    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-54d767fc5d-pv29k" podStartSLOduration=33.133345945 podStartE2EDuration="33.133345945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:46:08.132325245 +0000 UTC m=+167.161957151" watchObservedRunningTime="2026-09-05 19:46:08.133345945 +0000 UTC m=+167.162977841"
Sep 05 19:46:08.747964 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:08.747895    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:46:08.750534 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:08.750508    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:46:09.119001 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:09.118967    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:46:09.120501 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:09.120476    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-54d767fc5d-pv29k"
Sep 05 19:46:10.122823 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:10.122784    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wm7bd" event={"ID":"cd029360-95f2-45c6-9648-1fe5fe757822","Type":"ContainerStarted","Data":"a7a8f6d575af570510e328acca6ba1a0ff492a540661e07fd931649ba54654d8"}
Sep 05 19:46:10.144348 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:10.144297    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-wm7bd" podStartSLOduration=33.39193315 podStartE2EDuration="35.14428103s" totalImagesPullingTime="1.75234788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:46:10.143446474 +0000 UTC m=+169.173078384" watchObservedRunningTime="2026-09-05 19:46:10.14428103 +0000 UTC m=+169.173912934"
Sep 05 19:46:11.455839 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:11.455802    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:46:11.458159 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:11.458128    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a9b2788-2ab7-4c18-b80d-b5192c2909e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pgxbk\" (UID: \"7a9b2788-2ab7-4c18-b80d-b5192c2909e0\") " pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:46:11.686991 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:11.686937    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pgxbk"
Sep 05 19:46:11.960424 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:11.960369    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pgxbk"]
Sep 05 19:46:11.962838 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:46:11.962812    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9b2788_2ab7_4c18_b80d_b5192c2909e0.slice/crio-db8ced6c90ff9917be78ff1340684ee950dcf7c75a3d4dd611c15f3aa1438286 WatchSource:0}: Error finding container db8ced6c90ff9917be78ff1340684ee950dcf7c75a3d4dd611c15f3aa1438286: Status 404 returned error can't find the container with id db8ced6c90ff9917be78ff1340684ee950dcf7c75a3d4dd611c15f3aa1438286
Sep 05 19:46:12.129906 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:12.129866    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pgxbk" event={"ID":"7a9b2788-2ab7-4c18-b80d-b5192c2909e0","Type":"ContainerStarted","Data":"d45a3b791ae25e455b417b91369800af67c2f91526e0c484d39d47118776f03f"}
Sep 05 19:46:12.130078 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:12.129945    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pgxbk" event={"ID":"7a9b2788-2ab7-4c18-b80d-b5192c2909e0","Type":"ContainerStarted","Data":"db8ced6c90ff9917be78ff1340684ee950dcf7c75a3d4dd611c15f3aa1438286"}
Sep 05 19:46:12.428875 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:12.428842    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:46:13.135735 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:13.135649    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pgxbk" event={"ID":"7a9b2788-2ab7-4c18-b80d-b5192c2909e0","Type":"ContainerStarted","Data":"706d0b048990fb081da6b21fbdc2c3d9ac903f876d4b52d6517c6689d8b8805c"}
Sep 05 19:46:15.108431 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:15.108400    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zj8hk"
Sep 05 19:46:15.146386 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:15.146342    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pgxbk" event={"ID":"7a9b2788-2ab7-4c18-b80d-b5192c2909e0","Type":"ContainerStarted","Data":"18c823d4b5f409f27720982a7c8ec9b8825bc73675310ae552ff9e094d852f51"}
Sep 05 19:46:15.163599 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:15.163550    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pgxbk" podStartSLOduration=33.9379307 podStartE2EDuration="36.163534304s" totalImagesPullingTime="2.225603604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 19:45:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:46:15.162724144 +0000 UTC m=+174.192356071" watchObservedRunningTime="2026-09-05 19:46:15.163534304 +0000 UTC m=+174.193166209"
Sep 05 19:46:25.277730 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.277692    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-wj2f7"]
Sep 05 19:46:25.282275 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.282243    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wj2f7"
Sep 05 19:46:25.285125 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.285094    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 05 19:46:25.285241 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.285096    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 05 19:46:25.285328 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.285309    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 05 19:46:25.285440 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.285421    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-fnxth\""
Sep 05 19:46:25.290895 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.290868    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-vptgc"]
Sep 05 19:46:25.293858 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.293837    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc"
Sep 05 19:46:25.295074 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.295053    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-llgfz"]
Sep 05 19:46:25.296087 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.296068    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5k8rp\""
Sep 05 19:46:25.296185 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.296073    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 05 19:46:25.296267 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.296246    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 05 19:46:25.296364 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.296313    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 05 19:46:25.298025 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.298009    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-wj2f7"]
Sep 05 19:46:25.298121 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.298098    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.300061 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.300042    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 05 19:46:25.300333 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.300316    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 05 19:46:25.300666 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.300641    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4sdpt\""
Sep 05 19:46:25.300873 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.300689    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 05 19:46:25.307076 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.307052    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-vptgc"]
Sep 05 19:46:25.356405 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.356368    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-metrics-client-ca\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.356405 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.356400    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g2l5\" (UniqueName: \"kubernetes.io/projected/1342e506-7bb6-4af1-82cc-c1e8139193e1-kube-api-access-7g2l5\") pod \"openshift-state-metrics-657fd6c54-wj2f7\" (UID: \"1342e506-7bb6-4af1-82cc-c1e8139193e1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wj2f7"
Sep 05 19:46:25.356604 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.356418    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcjdg\" (UniqueName: \"kubernetes.io/projected/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-kube-api-access-wcjdg\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.356604 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.356441    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-node-exporter-wtmp\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.356604 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.356489    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-node-exporter-accelerators-collector-config\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.356604 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.356526    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ed7a58e-752e-4a92-a48b-6db480b280d3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-vptgc\" (UID: \"4ed7a58e-752e-4a92-a48b-6db480b280d3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc"
Sep 05 19:46:25.356604 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.356555    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ed7a58e-752e-4a92-a48b-6db480b280d3-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-vptgc\" (UID: \"4ed7a58e-752e-4a92-a48b-6db480b280d3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc"
Sep 05 19:46:25.356604 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.356576    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1342e506-7bb6-4af1-82cc-c1e8139193e1-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-wj2f7\" (UID: \"1342e506-7bb6-4af1-82cc-c1e8139193e1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wj2f7"
Sep 05 19:46:25.356604 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.356591    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-node-exporter-textfile\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.356877 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.356622    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxqv9\" (UniqueName: \"kubernetes.io/projected/4ed7a58e-752e-4a92-a48b-6db480b280d3-kube-api-access-pxqv9\") pod \"kube-state-metrics-74cb57df7-vptgc\" (UID: \"4ed7a58e-752e-4a92-a48b-6db480b280d3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc"
Sep 05 19:46:25.356877 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.356647    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1342e506-7bb6-4af1-82cc-c1e8139193e1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-wj2f7\" (UID: \"1342e506-7bb6-4af1-82cc-c1e8139193e1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wj2f7"
Sep 05 19:46:25.356877 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.356670    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.356877 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.356695    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4ed7a58e-752e-4a92-a48b-6db480b280d3-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-vptgc\" (UID: \"4ed7a58e-752e-4a92-a48b-6db480b280d3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc"
Sep 05 19:46:25.356877 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.356718    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ed7a58e-752e-4a92-a48b-6db480b280d3-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-vptgc\" (UID: \"4ed7a58e-752e-4a92-a48b-6db480b280d3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc"
Sep 05 19:46:25.356877 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.356742    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-sys\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.356877 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.356765    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1342e506-7bb6-4af1-82cc-c1e8139193e1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-wj2f7\" (UID: \"1342e506-7bb6-4af1-82cc-c1e8139193e1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wj2f7"
Sep 05 19:46:25.356877 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.356783    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-node-exporter-tls\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.356877 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.356810    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4ed7a58e-752e-4a92-a48b-6db480b280d3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-vptgc\" (UID: \"4ed7a58e-752e-4a92-a48b-6db480b280d3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc"
Sep 05 19:46:25.356877 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.356841    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-root\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.457847 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.457811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-node-exporter-tls\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.457847 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.457849    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4ed7a58e-752e-4a92-a48b-6db480b280d3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-vptgc\" (UID: \"4ed7a58e-752e-4a92-a48b-6db480b280d3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc"
Sep 05 19:46:25.458090 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.457878    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-root\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.458090 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.457902    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-metrics-client-ca\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.458090 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.457926    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g2l5\" (UniqueName: \"kubernetes.io/projected/1342e506-7bb6-4af1-82cc-c1e8139193e1-kube-api-access-7g2l5\") pod \"openshift-state-metrics-657fd6c54-wj2f7\" (UID: \"1342e506-7bb6-4af1-82cc-c1e8139193e1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wj2f7"
Sep 05 19:46:25.458090 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.457947    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcjdg\" (UniqueName: \"kubernetes.io/projected/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-kube-api-access-wcjdg\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.458090 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.457949    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-root\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.458090 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.457974    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-node-exporter-wtmp\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.458425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.458100    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-node-exporter-wtmp\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.458425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.458144    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-node-exporter-accelerators-collector-config\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.458425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.458190    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ed7a58e-752e-4a92-a48b-6db480b280d3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-vptgc\" (UID: \"4ed7a58e-752e-4a92-a48b-6db480b280d3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc"
Sep 05 19:46:25.458425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.458224    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ed7a58e-752e-4a92-a48b-6db480b280d3-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-vptgc\" (UID: \"4ed7a58e-752e-4a92-a48b-6db480b280d3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc"
Sep 05 19:46:25.458425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.458259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1342e506-7bb6-4af1-82cc-c1e8139193e1-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-wj2f7\" (UID: \"1342e506-7bb6-4af1-82cc-c1e8139193e1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wj2f7"
Sep 05 19:46:25.458425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.458297    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-node-exporter-textfile\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.458425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.458351    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxqv9\" (UniqueName: \"kubernetes.io/projected/4ed7a58e-752e-4a92-a48b-6db480b280d3-kube-api-access-pxqv9\") pod \"kube-state-metrics-74cb57df7-vptgc\" (UID: \"4ed7a58e-752e-4a92-a48b-6db480b280d3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc"
Sep 05 19:46:25.458425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.458378    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1342e506-7bb6-4af1-82cc-c1e8139193e1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-wj2f7\" (UID: \"1342e506-7bb6-4af1-82cc-c1e8139193e1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wj2f7"
Sep 05 19:46:25.458425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.458423    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.458868 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.458463    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4ed7a58e-752e-4a92-a48b-6db480b280d3-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-vptgc\" (UID: \"4ed7a58e-752e-4a92-a48b-6db480b280d3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc"
Sep 05 19:46:25.458868 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.458486    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ed7a58e-752e-4a92-a48b-6db480b280d3-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-vptgc\" (UID: \"4ed7a58e-752e-4a92-a48b-6db480b280d3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc"
Sep 05 19:46:25.458868 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.458512    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-sys\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.458868 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.458550    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1342e506-7bb6-4af1-82cc-c1e8139193e1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-wj2f7\" (UID: \"1342e506-7bb6-4af1-82cc-c1e8139193e1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wj2f7"
Sep 05 19:46:25.458868 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.458634    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-metrics-client-ca\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.458868 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.458705    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4ed7a58e-752e-4a92-a48b-6db480b280d3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-vptgc\" (UID: \"4ed7a58e-752e-4a92-a48b-6db480b280d3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc"
Sep 05 19:46:25.458868 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.458864    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-node-exporter-textfile\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.459257 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.458705    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-node-exporter-accelerators-collector-config\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.459257 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.459057    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-sys\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.459257 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.459192    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4ed7a58e-752e-4a92-a48b-6db480b280d3-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-vptgc\" (UID: \"4ed7a58e-752e-4a92-a48b-6db480b280d3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc"
Sep 05 19:46:25.459633 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.459604    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1342e506-7bb6-4af1-82cc-c1e8139193e1-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-wj2f7\" (UID: \"1342e506-7bb6-4af1-82cc-c1e8139193e1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wj2f7"
Sep 05 19:46:25.459633 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.459619    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ed7a58e-752e-4a92-a48b-6db480b280d3-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-vptgc\" (UID: \"4ed7a58e-752e-4a92-a48b-6db480b280d3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc"
Sep 05 19:46:25.461011 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.460986    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-node-exporter-tls\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.461380 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.461153    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1342e506-7bb6-4af1-82cc-c1e8139193e1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-wj2f7\" (UID: \"1342e506-7bb6-4af1-82cc-c1e8139193e1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wj2f7"
Sep 05 19:46:25.461380 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.461234    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.461555 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.461532    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ed7a58e-752e-4a92-a48b-6db480b280d3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-vptgc\" (UID: \"4ed7a58e-752e-4a92-a48b-6db480b280d3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc"
Sep 05 19:46:25.461611 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.461561    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ed7a58e-752e-4a92-a48b-6db480b280d3-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-vptgc\" (UID: \"4ed7a58e-752e-4a92-a48b-6db480b280d3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc"
Sep 05 19:46:25.461611 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.461541    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1342e506-7bb6-4af1-82cc-c1e8139193e1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-wj2f7\" (UID: \"1342e506-7bb6-4af1-82cc-c1e8139193e1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wj2f7"
Sep 05 19:46:25.465871 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.465849    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxqv9\" (UniqueName: \"kubernetes.io/projected/4ed7a58e-752e-4a92-a48b-6db480b280d3-kube-api-access-pxqv9\") pod \"kube-state-metrics-74cb57df7-vptgc\" (UID: \"4ed7a58e-752e-4a92-a48b-6db480b280d3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc"
Sep 05 19:46:25.466277 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.466256    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcjdg\" (UniqueName: \"kubernetes.io/projected/c82b3d00-c6ca-4e58-a4dc-e96f708cec4c-kube-api-access-wcjdg\") pod \"node-exporter-llgfz\" (UID: \"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c\") " pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.466351 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.466334    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g2l5\" (UniqueName: \"kubernetes.io/projected/1342e506-7bb6-4af1-82cc-c1e8139193e1-kube-api-access-7g2l5\") pod \"openshift-state-metrics-657fd6c54-wj2f7\" (UID: \"1342e506-7bb6-4af1-82cc-c1e8139193e1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wj2f7"
Sep 05 19:46:25.591491 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.591446    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wj2f7"
Sep 05 19:46:25.603922 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.603899    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc"
Sep 05 19:46:25.612290 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.612225    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-llgfz"
Sep 05 19:46:25.623283 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:46:25.623249    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82b3d00_c6ca_4e58_a4dc_e96f708cec4c.slice/crio-42fba09ccc4f8085994a6b57c3d585cf61f673be3501031261aeacb4f70a0b3b WatchSource:0}: Error finding container 42fba09ccc4f8085994a6b57c3d585cf61f673be3501031261aeacb4f70a0b3b: Status 404 returned error can't find the container with id 42fba09ccc4f8085994a6b57c3d585cf61f673be3501031261aeacb4f70a0b3b
Sep 05 19:46:25.870948 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.870878    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-wj2f7"]
Sep 05 19:46:25.873437 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:46:25.873410    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1342e506_7bb6_4af1_82cc_c1e8139193e1.slice/crio-3d864166cd0de569114b5d1618849a8658e48fb13266ebba86db7ef21af8c7b3 WatchSource:0}: Error finding container 3d864166cd0de569114b5d1618849a8658e48fb13266ebba86db7ef21af8c7b3: Status 404 returned error can't find the container with id 3d864166cd0de569114b5d1618849a8658e48fb13266ebba86db7ef21af8c7b3
Sep 05 19:46:25.893310 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:25.893221    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-vptgc"]
Sep 05 19:46:25.905086 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:46:25.905058    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed7a58e_752e_4a92_a48b_6db480b280d3.slice/crio-0819a4ab9b1ac317831e8045ae2b2001a17f875a1de9fc4de7d2873d5243bedc WatchSource:0}: Error finding container 0819a4ab9b1ac317831e8045ae2b2001a17f875a1de9fc4de7d2873d5243bedc: Status 404 returned error can't find the container with id 0819a4ab9b1ac317831e8045ae2b2001a17f875a1de9fc4de7d2873d5243bedc
Sep 05 19:46:26.184978 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:26.184882    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc" event={"ID":"4ed7a58e-752e-4a92-a48b-6db480b280d3","Type":"ContainerStarted","Data":"0819a4ab9b1ac317831e8045ae2b2001a17f875a1de9fc4de7d2873d5243bedc"}
Sep 05 19:46:26.187387 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:26.187357    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wj2f7" event={"ID":"1342e506-7bb6-4af1-82cc-c1e8139193e1","Type":"ContainerStarted","Data":"f87dd941e22c011880d1b64dd10fdd420eb1318138591d492f147101134d6cf2"}
Sep 05 19:46:26.187535 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:26.187421    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wj2f7" event={"ID":"1342e506-7bb6-4af1-82cc-c1e8139193e1","Type":"ContainerStarted","Data":"973695348a24c60e010e8ad8de1f874566416fd0f70208252dcdbaeb243a00cf"}
Sep 05 19:46:26.187535 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:26.187451    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wj2f7" event={"ID":"1342e506-7bb6-4af1-82cc-c1e8139193e1","Type":"ContainerStarted","Data":"3d864166cd0de569114b5d1618849a8658e48fb13266ebba86db7ef21af8c7b3"}
Sep 05 19:46:26.188721 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:26.188695    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-llgfz" event={"ID":"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c","Type":"ContainerStarted","Data":"42fba09ccc4f8085994a6b57c3d585cf61f673be3501031261aeacb4f70a0b3b"}
Sep 05 19:46:27.114290 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:27.114254    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-fd756675f-mnwfn"
Sep 05 19:46:27.195761 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:27.195668    2583 generic.go:356] "Generic (PLEG): container finished" podID="c82b3d00-c6ca-4e58-a4dc-e96f708cec4c" containerID="18902ea505cc281a9f54d38ba4bc8f0b5591cf55768feb03de61d9077d8c4602" exitCode=0
Sep 05 19:46:27.195761 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:27.195718    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-llgfz" event={"ID":"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c","Type":"ContainerDied","Data":"18902ea505cc281a9f54d38ba4bc8f0b5591cf55768feb03de61d9077d8c4602"}
Sep 05 19:46:28.201386 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:28.201346    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wj2f7" event={"ID":"1342e506-7bb6-4af1-82cc-c1e8139193e1","Type":"ContainerStarted","Data":"e1ed435d4c8fccfa75f0897d0cd4bd3ac4e21b828d035cc486386ef58927f093"}
Sep 05 19:46:28.203366 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:28.203344    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-llgfz" event={"ID":"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c","Type":"ContainerStarted","Data":"f26f1750997c41c6f0c09f019426b6d55538bac90293e33548289a04f6e9a8d7"}
Sep 05 19:46:28.203458 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:28.203391    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-llgfz" event={"ID":"c82b3d00-c6ca-4e58-a4dc-e96f708cec4c","Type":"ContainerStarted","Data":"9b02ce6ea05219e4ae0a96ac79eae71f6b3c818c73b8d375cfcebc704f434dfc"}
Sep 05 19:46:28.205356 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:28.205335    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc" event={"ID":"4ed7a58e-752e-4a92-a48b-6db480b280d3","Type":"ContainerStarted","Data":"e6cb4a4cf8251588c1123a151405787521ff6f629c2d93e45031c0b1ef1af0eb"}
Sep 05 19:46:28.205458 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:28.205372    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc" event={"ID":"4ed7a58e-752e-4a92-a48b-6db480b280d3","Type":"ContainerStarted","Data":"030943fed8a63e919a1b89b680299003879d839f4a2ec7cf21694a83d322cbf3"}
Sep 05 19:46:28.205458 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:28.205390    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc" event={"ID":"4ed7a58e-752e-4a92-a48b-6db480b280d3","Type":"ContainerStarted","Data":"9e78767466e5403b6456029ebaf39c36a2de701034a9bc5bc707e61e0fd47f51"}
Sep 05 19:46:28.230672 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:28.230620    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-wj2f7" podStartSLOduration=1.7639080759999999 podStartE2EDuration="3.230603399s" totalImagesPullingTime="1.466695323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:46:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:46:28.230146783 +0000 UTC m=+187.259778688" watchObservedRunningTime="2026-09-05 19:46:28.230603399 +0000 UTC m=+187.260235304"
Sep 05 19:46:28.282979 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:28.282933    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-vptgc" podStartSLOduration=1.7426169900000001 podStartE2EDuration="3.282917407s" totalImagesPullingTime="1.540300417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:46:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:46:28.282491258 +0000 UTC m=+187.312123163" watchObservedRunningTime="2026-09-05 19:46:28.282917407 +0000 UTC m=+187.312549312"
Sep 05 19:46:28.301968 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:28.301084    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-llgfz" podStartSLOduration=2.266997423 podStartE2EDuration="3.301064407s" totalImagesPullingTime="959.653474ms" totalInitContainerRuntime="74.41351ms" isStatefulPod=true podCreationTimestamp="2026-09-05 19:46:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:46:28.301065758 +0000 UTC m=+187.330697685" watchObservedRunningTime="2026-09-05 19:46:28.301064407 +0000 UTC m=+187.330696312"
Sep 05 19:46:50.283463 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:50.283377    2583 generic.go:356] "Generic (PLEG): container finished" podID="5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88" containerID="02bc1e6089e6ec38353873e9525d3f9d63d8d5c79ecfe41744b4a757fd9865c7" exitCode=0
Sep 05 19:46:50.283463 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:50.283431    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-4v2k6" event={"ID":"5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88","Type":"ContainerDied","Data":"02bc1e6089e6ec38353873e9525d3f9d63d8d5c79ecfe41744b4a757fd9865c7"}
Sep 05 19:46:50.283938 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:50.283719    2583 scope.go:117] "RemoveContainer" containerID="02bc1e6089e6ec38353873e9525d3f9d63d8d5c79ecfe41744b4a757fd9865c7"
Sep 05 19:46:51.289431 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:46:51.289388    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-4v2k6" event={"ID":"5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88","Type":"ContainerStarted","Data":"7c0969b51f6f9b01e2c7dd825ad7b80597fdd7995cc908346e83fc01313a5969"}
Sep 05 19:47:32.285659 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:47:32.285620    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs\") pod \"network-metrics-daemon-rwwbl\" (UID: \"b9720241-ac7b-4d79-9644-233caff66a8c\") " pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:47:32.287870 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:47:32.287847    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9720241-ac7b-4d79-9644-233caff66a8c-metrics-certs\") pod \"network-metrics-daemon-rwwbl\" (UID: \"b9720241-ac7b-4d79-9644-233caff66a8c\") " pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:47:32.532238 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:47:32.532204    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-srspc\""
Sep 05 19:47:32.540293 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:47:32.540222    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rwwbl"
Sep 05 19:47:32.810217 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:47:32.810190    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rwwbl"]
Sep 05 19:47:32.812628 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:47:32.812603    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9720241_ac7b_4d79_9644_233caff66a8c.slice/crio-b9ec54841d83a0c080f69e5d6096b923fd682691c0d556e8d197be1eb4344a0c WatchSource:0}: Error finding container b9ec54841d83a0c080f69e5d6096b923fd682691c0d556e8d197be1eb4344a0c: Status 404 returned error can't find the container with id b9ec54841d83a0c080f69e5d6096b923fd682691c0d556e8d197be1eb4344a0c
Sep 05 19:47:33.436615 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:47:33.436579    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rwwbl" event={"ID":"b9720241-ac7b-4d79-9644-233caff66a8c","Type":"ContainerStarted","Data":"b9ec54841d83a0c080f69e5d6096b923fd682691c0d556e8d197be1eb4344a0c"}
Sep 05 19:47:34.442670 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:47:34.442636    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rwwbl" event={"ID":"b9720241-ac7b-4d79-9644-233caff66a8c","Type":"ContainerStarted","Data":"21c0cfa1f029f67a7a139e5c80ec7de482647dd50ac272dde81150773fdff0f3"}
Sep 05 19:47:34.443149 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:47:34.442694    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rwwbl" event={"ID":"b9720241-ac7b-4d79-9644-233caff66a8c","Type":"ContainerStarted","Data":"2784eefb5cbfc31bb3781bc223711b1bbbafa066c09311d113780a5ed75201b9"}
Sep 05 19:47:34.458462 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:47:34.458419    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rwwbl" podStartSLOduration=252.562470328 podStartE2EDuration="4m13.458400606s" totalImagesPullingTime="895.930278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:47:34.457934219 +0000 UTC m=+253.487566124" watchObservedRunningTime="2026-09-05 19:47:34.458400606 +0000 UTC m=+253.488032505"
Sep 05 19:48:21.346506 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:48:21.346481    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf9tj_f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b/ovn-acl-logging/0.log"
Sep 05 19:48:21.348944 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:48:21.348917    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf9tj_f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b/ovn-acl-logging/0.log"
Sep 05 19:48:21.350308 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:48:21.350287    2583 kubelet.go:1648] "Image garbage collection succeeded"
Sep 05 19:49:42.271905 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:49:42.271829    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc4495bbf-w2vhb"]
Sep 05 19:49:42.274897 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:49:42.274880    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc4495bbf-w2vhb"
Sep 05 19:49:42.277373 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:49:42.277349    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 05 19:49:42.277490 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:49:42.277454    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 05 19:49:42.277490 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:49:42.277463    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 05 19:49:42.277585 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:49:42.277489    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 05 19:49:42.284514 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:49:42.284492    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc4495bbf-w2vhb"]
Sep 05 19:49:42.451398 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:49:42.451367    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b5540470-44e3-48c9-becd-d589034696cf-klusterlet-config\") pod \"klusterlet-addon-workmgr-7fc4495bbf-w2vhb\" (UID: \"b5540470-44e3-48c9-becd-d589034696cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc4495bbf-w2vhb"
Sep 05 19:49:42.451559 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:49:42.451409    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sjck\" (UniqueName: \"kubernetes.io/projected/b5540470-44e3-48c9-becd-d589034696cf-kube-api-access-8sjck\") pod \"klusterlet-addon-workmgr-7fc4495bbf-w2vhb\" (UID: \"b5540470-44e3-48c9-becd-d589034696cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc4495bbf-w2vhb"
Sep 05 19:49:42.451559 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:49:42.451484    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5540470-44e3-48c9-becd-d589034696cf-tmp\") pod \"klusterlet-addon-workmgr-7fc4495bbf-w2vhb\" (UID: \"b5540470-44e3-48c9-becd-d589034696cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc4495bbf-w2vhb"
Sep 05 19:49:42.551876 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:49:42.551789    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sjck\" (UniqueName: \"kubernetes.io/projected/b5540470-44e3-48c9-becd-d589034696cf-kube-api-access-8sjck\") pod \"klusterlet-addon-workmgr-7fc4495bbf-w2vhb\" (UID: \"b5540470-44e3-48c9-becd-d589034696cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc4495bbf-w2vhb"
Sep 05 19:49:42.551876 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:49:42.551835    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5540470-44e3-48c9-becd-d589034696cf-tmp\") pod \"klusterlet-addon-workmgr-7fc4495bbf-w2vhb\" (UID: \"b5540470-44e3-48c9-becd-d589034696cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc4495bbf-w2vhb"
Sep 05 19:49:42.552093 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:49:42.552064    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b5540470-44e3-48c9-becd-d589034696cf-klusterlet-config\") pod \"klusterlet-addon-workmgr-7fc4495bbf-w2vhb\" (UID: \"b5540470-44e3-48c9-becd-d589034696cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc4495bbf-w2vhb"
Sep 05 19:49:42.552307 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:49:42.552273    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5540470-44e3-48c9-becd-d589034696cf-tmp\") pod \"klusterlet-addon-workmgr-7fc4495bbf-w2vhb\" (UID: \"b5540470-44e3-48c9-becd-d589034696cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc4495bbf-w2vhb"
Sep 05 19:49:42.554527 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:49:42.554508    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b5540470-44e3-48c9-becd-d589034696cf-klusterlet-config\") pod \"klusterlet-addon-workmgr-7fc4495bbf-w2vhb\" (UID: \"b5540470-44e3-48c9-becd-d589034696cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc4495bbf-w2vhb"
Sep 05 19:49:42.560971 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:49:42.560938    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sjck\" (UniqueName: \"kubernetes.io/projected/b5540470-44e3-48c9-becd-d589034696cf-kube-api-access-8sjck\") pod \"klusterlet-addon-workmgr-7fc4495bbf-w2vhb\" (UID: \"b5540470-44e3-48c9-becd-d589034696cf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc4495bbf-w2vhb"
Sep 05 19:49:42.583949 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:49:42.583909    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc4495bbf-w2vhb"
Sep 05 19:49:42.851922 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:49:42.851889    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc4495bbf-w2vhb"]
Sep 05 19:49:42.855500 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:49:42.855469    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5540470_44e3_48c9_becd_d589034696cf.slice/crio-f86e44b2a4ff9c441989fbb3bd21f6a6afdb240502bf231ae49204c3cba3ce67 WatchSource:0}: Error finding container f86e44b2a4ff9c441989fbb3bd21f6a6afdb240502bf231ae49204c3cba3ce67: Status 404 returned error can't find the container with id f86e44b2a4ff9c441989fbb3bd21f6a6afdb240502bf231ae49204c3cba3ce67
Sep 05 19:49:42.857297 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:49:42.857280    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 19:49:42.877905 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:49:42.877881    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc4495bbf-w2vhb" event={"ID":"b5540470-44e3-48c9-becd-d589034696cf","Type":"ContainerStarted","Data":"f86e44b2a4ff9c441989fbb3bd21f6a6afdb240502bf231ae49204c3cba3ce67"}
Sep 05 19:49:48.906738 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:49:48.906695    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc4495bbf-w2vhb" event={"ID":"b5540470-44e3-48c9-becd-d589034696cf","Type":"ContainerStarted","Data":"4bab8974f28bd05a740a49ef07f1c1117e4eeee3a7923a4de226662cade35c57"}
Sep 05 19:49:48.907161 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:49:48.906911    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc4495bbf-w2vhb"
Sep 05 19:49:48.908893 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:49:48.908870    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc4495bbf-w2vhb"
Sep 05 19:49:48.922853 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:49:48.922810    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fc4495bbf-w2vhb" podStartSLOduration=1.58743328 podStartE2EDuration="6.922796516s" totalImagesPullingTime="5.335363236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:49:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:49:48.921900269 +0000 UTC m=+387.951532174" watchObservedRunningTime="2026-09-05 19:49:48.922796516 +0000 UTC m=+387.952428421"
Sep 05 19:50:12.693560 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:12.693524    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q"]
Sep 05 19:50:12.697735 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:12.697714    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q"
Sep 05 19:50:12.699923 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:12.699898    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 05 19:50:12.700315 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:12.700299    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 05 19:50:12.700315 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:12.700307    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vvg94\""
Sep 05 19:50:12.704726 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:12.704703    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q"]
Sep 05 19:50:12.783913 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:12.783869    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd6c93c7-cf44-4e19-b78a-477c47a76c9c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q\" (UID: \"dd6c93c7-cf44-4e19-b78a-477c47a76c9c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q"
Sep 05 19:50:12.784098 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:12.783956    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bgt4\" (UniqueName: \"kubernetes.io/projected/dd6c93c7-cf44-4e19-b78a-477c47a76c9c-kube-api-access-4bgt4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q\" (UID: \"dd6c93c7-cf44-4e19-b78a-477c47a76c9c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q"
Sep 05 19:50:12.784098 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:12.784009    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd6c93c7-cf44-4e19-b78a-477c47a76c9c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q\" (UID: \"dd6c93c7-cf44-4e19-b78a-477c47a76c9c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q"
Sep 05 19:50:12.885395 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:12.885358    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd6c93c7-cf44-4e19-b78a-477c47a76c9c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q\" (UID: \"dd6c93c7-cf44-4e19-b78a-477c47a76c9c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q"
Sep 05 19:50:12.885549 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:12.885405    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bgt4\" (UniqueName: \"kubernetes.io/projected/dd6c93c7-cf44-4e19-b78a-477c47a76c9c-kube-api-access-4bgt4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q\" (UID: \"dd6c93c7-cf44-4e19-b78a-477c47a76c9c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q"
Sep 05 19:50:12.885549 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:12.885441    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd6c93c7-cf44-4e19-b78a-477c47a76c9c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q\" (UID: \"dd6c93c7-cf44-4e19-b78a-477c47a76c9c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q"
Sep 05 19:50:12.885854 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:12.885834    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd6c93c7-cf44-4e19-b78a-477c47a76c9c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q\" (UID: \"dd6c93c7-cf44-4e19-b78a-477c47a76c9c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q"
Sep 05 19:50:12.885890 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:12.885833    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd6c93c7-cf44-4e19-b78a-477c47a76c9c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q\" (UID: \"dd6c93c7-cf44-4e19-b78a-477c47a76c9c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q"
Sep 05 19:50:12.893858 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:12.893827    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bgt4\" (UniqueName: \"kubernetes.io/projected/dd6c93c7-cf44-4e19-b78a-477c47a76c9c-kube-api-access-4bgt4\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q\" (UID: \"dd6c93c7-cf44-4e19-b78a-477c47a76c9c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q"
Sep 05 19:50:13.007476 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:13.007393    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q"
Sep 05 19:50:13.279877 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:13.279853    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q"]
Sep 05 19:50:13.282157 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:50:13.282130    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6c93c7_cf44_4e19_b78a_477c47a76c9c.slice/crio-fab6014dda6ac60ffafd426d50b254006bfe903fd59ad27b93a7780cfd0faa49 WatchSource:0}: Error finding container fab6014dda6ac60ffafd426d50b254006bfe903fd59ad27b93a7780cfd0faa49: Status 404 returned error can't find the container with id fab6014dda6ac60ffafd426d50b254006bfe903fd59ad27b93a7780cfd0faa49
Sep 05 19:50:13.994414 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:13.994378    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q" event={"ID":"dd6c93c7-cf44-4e19-b78a-477c47a76c9c","Type":"ContainerStarted","Data":"fab6014dda6ac60ffafd426d50b254006bfe903fd59ad27b93a7780cfd0faa49"}
Sep 05 19:50:20.019638 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:20.019603    2583 generic.go:356] "Generic (PLEG): container finished" podID="dd6c93c7-cf44-4e19-b78a-477c47a76c9c" containerID="d56d5b17d2f03e1fad32aa553395ad3a33e42b6fcda5ec53a017e0adc9048e5e" exitCode=0
Sep 05 19:50:20.019990 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:20.019697    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q" event={"ID":"dd6c93c7-cf44-4e19-b78a-477c47a76c9c","Type":"ContainerDied","Data":"d56d5b17d2f03e1fad32aa553395ad3a33e42b6fcda5ec53a017e0adc9048e5e"}
Sep 05 19:50:23.032299 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:23.032264    2583 generic.go:356] "Generic (PLEG): container finished" podID="dd6c93c7-cf44-4e19-b78a-477c47a76c9c" containerID="e92da27dcaa1ad122a078d6e9474ca8989a46543f9b8d800063a54112e6a2496" exitCode=0
Sep 05 19:50:23.032648 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:23.032346    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q" event={"ID":"dd6c93c7-cf44-4e19-b78a-477c47a76c9c","Type":"ContainerDied","Data":"e92da27dcaa1ad122a078d6e9474ca8989a46543f9b8d800063a54112e6a2496"}
Sep 05 19:50:30.067781 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:30.067746    2583 generic.go:356] "Generic (PLEG): container finished" podID="dd6c93c7-cf44-4e19-b78a-477c47a76c9c" containerID="c1f9fea22212824d8cec89fe70ea8f6065ce648b5a1d8778d7732d634e10a9e7" exitCode=0
Sep 05 19:50:30.068118 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:30.067785    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q" event={"ID":"dd6c93c7-cf44-4e19-b78a-477c47a76c9c","Type":"ContainerDied","Data":"c1f9fea22212824d8cec89fe70ea8f6065ce648b5a1d8778d7732d634e10a9e7"}
Sep 05 19:50:31.191633 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:31.191610    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q"
Sep 05 19:50:31.337753 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:31.337666    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd6c93c7-cf44-4e19-b78a-477c47a76c9c-util\") pod \"dd6c93c7-cf44-4e19-b78a-477c47a76c9c\" (UID: \"dd6c93c7-cf44-4e19-b78a-477c47a76c9c\") "
Sep 05 19:50:31.337753 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:31.337708    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd6c93c7-cf44-4e19-b78a-477c47a76c9c-bundle\") pod \"dd6c93c7-cf44-4e19-b78a-477c47a76c9c\" (UID: \"dd6c93c7-cf44-4e19-b78a-477c47a76c9c\") "
Sep 05 19:50:31.337982 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:31.337774    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bgt4\" (UniqueName: \"kubernetes.io/projected/dd6c93c7-cf44-4e19-b78a-477c47a76c9c-kube-api-access-4bgt4\") pod \"dd6c93c7-cf44-4e19-b78a-477c47a76c9c\" (UID: \"dd6c93c7-cf44-4e19-b78a-477c47a76c9c\") "
Sep 05 19:50:31.338413 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:31.338388    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd6c93c7-cf44-4e19-b78a-477c47a76c9c-bundle" (OuterVolumeSpecName: "bundle") pod "dd6c93c7-cf44-4e19-b78a-477c47a76c9c" (UID: "dd6c93c7-cf44-4e19-b78a-477c47a76c9c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:50:31.339946 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:31.339921    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd6c93c7-cf44-4e19-b78a-477c47a76c9c-kube-api-access-4bgt4" (OuterVolumeSpecName: "kube-api-access-4bgt4") pod "dd6c93c7-cf44-4e19-b78a-477c47a76c9c" (UID: "dd6c93c7-cf44-4e19-b78a-477c47a76c9c"). InnerVolumeSpecName "kube-api-access-4bgt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:50:31.341726 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:31.341702    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd6c93c7-cf44-4e19-b78a-477c47a76c9c-util" (OuterVolumeSpecName: "util") pod "dd6c93c7-cf44-4e19-b78a-477c47a76c9c" (UID: "dd6c93c7-cf44-4e19-b78a-477c47a76c9c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:50:31.438960 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:31.438926    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bgt4\" (UniqueName: \"kubernetes.io/projected/dd6c93c7-cf44-4e19-b78a-477c47a76c9c-kube-api-access-4bgt4\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:50:31.438960 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:31.438953    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd6c93c7-cf44-4e19-b78a-477c47a76c9c-util\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:50:31.439144 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:31.438967    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd6c93c7-cf44-4e19-b78a-477c47a76c9c-bundle\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:50:32.078258 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:32.078223    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q" event={"ID":"dd6c93c7-cf44-4e19-b78a-477c47a76c9c","Type":"ContainerDied","Data":"fab6014dda6ac60ffafd426d50b254006bfe903fd59ad27b93a7780cfd0faa49"}
Sep 05 19:50:32.078415 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:32.078276    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fab6014dda6ac60ffafd426d50b254006bfe903fd59ad27b93a7780cfd0faa49"
Sep 05 19:50:32.078415 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:32.078238    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407jn2q"
Sep 05 19:50:35.647970 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:35.647939    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rg6t9"]
Sep 05 19:50:35.648361 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:35.648213    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd6c93c7-cf44-4e19-b78a-477c47a76c9c" containerName="pull"
Sep 05 19:50:35.648361 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:35.648224    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd6c93c7-cf44-4e19-b78a-477c47a76c9c" containerName="pull"
Sep 05 19:50:35.648361 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:35.648235    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd6c93c7-cf44-4e19-b78a-477c47a76c9c" containerName="util"
Sep 05 19:50:35.648361 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:35.648242    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd6c93c7-cf44-4e19-b78a-477c47a76c9c" containerName="util"
Sep 05 19:50:35.648361 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:35.648249    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd6c93c7-cf44-4e19-b78a-477c47a76c9c" containerName="extract"
Sep 05 19:50:35.648361 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:35.648254    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd6c93c7-cf44-4e19-b78a-477c47a76c9c" containerName="extract"
Sep 05 19:50:35.648361 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:35.648308    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd6c93c7-cf44-4e19-b78a-477c47a76c9c" containerName="extract"
Sep 05 19:50:35.692087 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:35.692056    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rg6t9"]
Sep 05 19:50:35.692252 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:35.692168    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rg6t9"
Sep 05 19:50:35.694758 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:35.694734    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-qplbk\""
Sep 05 19:50:35.694857 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:35.694766    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 05 19:50:35.694857 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:35.694734    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 05 19:50:35.770340 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:35.770309    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0305c8ee-af7e-4f7f-842c-b3fec7576856-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-rg6t9\" (UID: \"0305c8ee-af7e-4f7f-842c-b3fec7576856\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rg6t9"
Sep 05 19:50:35.770495 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:35.770358    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr99j\" (UniqueName: \"kubernetes.io/projected/0305c8ee-af7e-4f7f-842c-b3fec7576856-kube-api-access-fr99j\") pod \"cert-manager-operator-controller-manager-55c444bf86-rg6t9\" (UID: \"0305c8ee-af7e-4f7f-842c-b3fec7576856\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rg6t9"
Sep 05 19:50:35.871000 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:35.870957    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0305c8ee-af7e-4f7f-842c-b3fec7576856-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-rg6t9\" (UID: \"0305c8ee-af7e-4f7f-842c-b3fec7576856\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rg6t9"
Sep 05 19:50:35.871000 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:35.871008    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr99j\" (UniqueName: \"kubernetes.io/projected/0305c8ee-af7e-4f7f-842c-b3fec7576856-kube-api-access-fr99j\") pod \"cert-manager-operator-controller-manager-55c444bf86-rg6t9\" (UID: \"0305c8ee-af7e-4f7f-842c-b3fec7576856\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rg6t9"
Sep 05 19:50:35.871417 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:35.871390    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0305c8ee-af7e-4f7f-842c-b3fec7576856-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-rg6t9\" (UID: \"0305c8ee-af7e-4f7f-842c-b3fec7576856\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rg6t9"
Sep 05 19:50:35.879398 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:35.879374    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr99j\" (UniqueName: \"kubernetes.io/projected/0305c8ee-af7e-4f7f-842c-b3fec7576856-kube-api-access-fr99j\") pod \"cert-manager-operator-controller-manager-55c444bf86-rg6t9\" (UID: \"0305c8ee-af7e-4f7f-842c-b3fec7576856\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rg6t9"
Sep 05 19:50:36.001147 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:36.001040    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rg6t9"
Sep 05 19:50:36.295234 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:36.295205    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rg6t9"]
Sep 05 19:50:36.298018 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:50:36.297991    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0305c8ee_af7e_4f7f_842c_b3fec7576856.slice/crio-148bdbacf465adf84f8bcdc7b7f73101eb33b569a305f930ee06bf7503968c0f WatchSource:0}: Error finding container 148bdbacf465adf84f8bcdc7b7f73101eb33b569a305f930ee06bf7503968c0f: Status 404 returned error can't find the container with id 148bdbacf465adf84f8bcdc7b7f73101eb33b569a305f930ee06bf7503968c0f
Sep 05 19:50:37.099523 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:37.099485    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rg6t9" event={"ID":"0305c8ee-af7e-4f7f-842c-b3fec7576856","Type":"ContainerStarted","Data":"148bdbacf465adf84f8bcdc7b7f73101eb33b569a305f930ee06bf7503968c0f"}
Sep 05 19:50:39.109096 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:39.109056    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rg6t9" event={"ID":"0305c8ee-af7e-4f7f-842c-b3fec7576856","Type":"ContainerStarted","Data":"0962d1563053b011e22f998f86ad2f28d0db13df51bcff120f0c2b023c0c0400"}
Sep 05 19:50:39.131613 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:39.131556    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rg6t9" podStartSLOduration=1.833311867 podStartE2EDuration="4.131537243s" totalImagesPullingTime="2.298225376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:50:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:50:39.131217111 +0000 UTC m=+438.160849017" watchObservedRunningTime="2026-09-05 19:50:39.131537243 +0000 UTC m=+438.161169148"
Sep 05 19:50:40.193647 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:40.193615    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp"]
Sep 05 19:50:40.195689 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:40.195669    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp"
Sep 05 19:50:40.197908 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:40.197888    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 05 19:50:40.198333 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:40.198317    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 05 19:50:40.198389 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:40.198317    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vvg94\""
Sep 05 19:50:40.206368 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:40.206345    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp"]
Sep 05 19:50:40.307733 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:40.307696    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1537c346-b2c8-4e9c-9aa0-6ede348bff59-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp\" (UID: \"1537c346-b2c8-4e9c-9aa0-6ede348bff59\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp"
Sep 05 19:50:40.307733 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:40.307738    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1537c346-b2c8-4e9c-9aa0-6ede348bff59-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp\" (UID: \"1537c346-b2c8-4e9c-9aa0-6ede348bff59\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp"
Sep 05 19:50:40.307951 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:40.307763    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzr4g\" (UniqueName: \"kubernetes.io/projected/1537c346-b2c8-4e9c-9aa0-6ede348bff59-kube-api-access-tzr4g\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp\" (UID: \"1537c346-b2c8-4e9c-9aa0-6ede348bff59\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp"
Sep 05 19:50:40.408412 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:40.408353    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1537c346-b2c8-4e9c-9aa0-6ede348bff59-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp\" (UID: \"1537c346-b2c8-4e9c-9aa0-6ede348bff59\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp"
Sep 05 19:50:40.408571 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:40.408437    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1537c346-b2c8-4e9c-9aa0-6ede348bff59-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp\" (UID: \"1537c346-b2c8-4e9c-9aa0-6ede348bff59\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp"
Sep 05 19:50:40.408571 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:40.408475    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzr4g\" (UniqueName: \"kubernetes.io/projected/1537c346-b2c8-4e9c-9aa0-6ede348bff59-kube-api-access-tzr4g\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp\" (UID: \"1537c346-b2c8-4e9c-9aa0-6ede348bff59\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp"
Sep 05 19:50:40.408812 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:40.408792    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1537c346-b2c8-4e9c-9aa0-6ede348bff59-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp\" (UID: \"1537c346-b2c8-4e9c-9aa0-6ede348bff59\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp"
Sep 05 19:50:40.408848 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:40.408822    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1537c346-b2c8-4e9c-9aa0-6ede348bff59-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp\" (UID: \"1537c346-b2c8-4e9c-9aa0-6ede348bff59\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp"
Sep 05 19:50:40.416789 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:40.416755    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzr4g\" (UniqueName: \"kubernetes.io/projected/1537c346-b2c8-4e9c-9aa0-6ede348bff59-kube-api-access-tzr4g\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp\" (UID: \"1537c346-b2c8-4e9c-9aa0-6ede348bff59\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp"
Sep 05 19:50:40.504752 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:40.504659    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp"
Sep 05 19:50:40.773433 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:40.773403    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp"]
Sep 05 19:50:40.775784 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:50:40.775754    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1537c346_b2c8_4e9c_9aa0_6ede348bff59.slice/crio-cd4bbd54c450ad914b33295de5cceab5d72d7831b919748ef755e863e11b2844 WatchSource:0}: Error finding container cd4bbd54c450ad914b33295de5cceab5d72d7831b919748ef755e863e11b2844: Status 404 returned error can't find the container with id cd4bbd54c450ad914b33295de5cceab5d72d7831b919748ef755e863e11b2844
Sep 05 19:50:41.118410 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:41.118370    2583 generic.go:356] "Generic (PLEG): container finished" podID="1537c346-b2c8-4e9c-9aa0-6ede348bff59" containerID="d089a1ae6fb138d0890b9a70ff85134fb34d93b8dbfae235914eb66f9c572f43" exitCode=0
Sep 05 19:50:41.118564 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:41.118423    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp" event={"ID":"1537c346-b2c8-4e9c-9aa0-6ede348bff59","Type":"ContainerDied","Data":"d089a1ae6fb138d0890b9a70ff85134fb34d93b8dbfae235914eb66f9c572f43"}
Sep 05 19:50:41.118564 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:41.118458    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp" event={"ID":"1537c346-b2c8-4e9c-9aa0-6ede348bff59","Type":"ContainerStarted","Data":"cd4bbd54c450ad914b33295de5cceab5d72d7831b919748ef755e863e11b2844"}
Sep 05 19:50:42.170175 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:42.170140    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jsgg5"]
Sep 05 19:50:42.172549 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:42.172528    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jsgg5"
Sep 05 19:50:42.174833 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:42.174801    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 05 19:50:42.175093 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:42.174800    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-nphnp\""
Sep 05 19:50:42.175214 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:42.175135    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 05 19:50:42.180372 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:42.180334    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jsgg5"]
Sep 05 19:50:42.323533 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:42.323498    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fjtn\" (UniqueName: \"kubernetes.io/projected/322217b0-5fd3-43c0-8485-8343e723661d-kube-api-access-5fjtn\") pod \"cert-manager-cainjector-559cdd8c87-jsgg5\" (UID: \"322217b0-5fd3-43c0-8485-8343e723661d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jsgg5"
Sep 05 19:50:42.323712 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:42.323561    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/322217b0-5fd3-43c0-8485-8343e723661d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jsgg5\" (UID: \"322217b0-5fd3-43c0-8485-8343e723661d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jsgg5"
Sep 05 19:50:42.425021 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:42.424934    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fjtn\" (UniqueName: \"kubernetes.io/projected/322217b0-5fd3-43c0-8485-8343e723661d-kube-api-access-5fjtn\") pod \"cert-manager-cainjector-559cdd8c87-jsgg5\" (UID: \"322217b0-5fd3-43c0-8485-8343e723661d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jsgg5"
Sep 05 19:50:42.425021 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:42.425014    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/322217b0-5fd3-43c0-8485-8343e723661d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jsgg5\" (UID: \"322217b0-5fd3-43c0-8485-8343e723661d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jsgg5"
Sep 05 19:50:42.433802 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:42.433714    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/322217b0-5fd3-43c0-8485-8343e723661d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jsgg5\" (UID: \"322217b0-5fd3-43c0-8485-8343e723661d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jsgg5"
Sep 05 19:50:42.434012 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:42.433988    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fjtn\" (UniqueName: \"kubernetes.io/projected/322217b0-5fd3-43c0-8485-8343e723661d-kube-api-access-5fjtn\") pod \"cert-manager-cainjector-559cdd8c87-jsgg5\" (UID: \"322217b0-5fd3-43c0-8485-8343e723661d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jsgg5"
Sep 05 19:50:42.488857 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:42.488816    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jsgg5"
Sep 05 19:50:42.769046 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:42.766924    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jsgg5"]
Sep 05 19:50:43.362747 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:50:43.362709    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod322217b0_5fd3_43c0_8485_8343e723661d.slice/crio-770b903a3cb96f413f303ac85f8a851d89adc846ae60e944e3d61306c257f763 WatchSource:0}: Error finding container 770b903a3cb96f413f303ac85f8a851d89adc846ae60e944e3d61306c257f763: Status 404 returned error can't find the container with id 770b903a3cb96f413f303ac85f8a851d89adc846ae60e944e3d61306c257f763
Sep 05 19:50:44.134386 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:44.134344    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jsgg5" event={"ID":"322217b0-5fd3-43c0-8485-8343e723661d","Type":"ContainerStarted","Data":"770b903a3cb96f413f303ac85f8a851d89adc846ae60e944e3d61306c257f763"}
Sep 05 19:50:44.136487 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:44.136451    2583 generic.go:356] "Generic (PLEG): container finished" podID="1537c346-b2c8-4e9c-9aa0-6ede348bff59" containerID="a8ea7b1c01e2a2812cb1b9738018b499dbaaa1fb335ef32842095608fbd79e40" exitCode=0
Sep 05 19:50:44.136618 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:44.136495    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp" event={"ID":"1537c346-b2c8-4e9c-9aa0-6ede348bff59","Type":"ContainerDied","Data":"a8ea7b1c01e2a2812cb1b9738018b499dbaaa1fb335ef32842095608fbd79e40"}
Sep 05 19:50:45.144660 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:45.144556    2583 generic.go:356] "Generic (PLEG): container finished" podID="1537c346-b2c8-4e9c-9aa0-6ede348bff59" containerID="cc4205e01958cab6072a3c1a6360c73bde6fb502da49b60d5e3ad4b31f365516" exitCode=0
Sep 05 19:50:45.144660 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:45.144644    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp" event={"ID":"1537c346-b2c8-4e9c-9aa0-6ede348bff59","Type":"ContainerDied","Data":"cc4205e01958cab6072a3c1a6360c73bde6fb502da49b60d5e3ad4b31f365516"}
Sep 05 19:50:46.150891 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:46.150802    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jsgg5" event={"ID":"322217b0-5fd3-43c0-8485-8343e723661d","Type":"ContainerStarted","Data":"855268ac6cefcc29f2ed4db3c78b136910e4faf897a72ee04d62f1f48ac96f75"}
Sep 05 19:50:46.169192 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:46.169104    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jsgg5" podStartSLOduration=1.639264893 podStartE2EDuration="4.16908397s" totalImagesPullingTime="2.529819077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:50:46.166698003 +0000 UTC m=+445.196329909" watchObservedRunningTime="2026-09-05 19:50:46.16908397 +0000 UTC m=+445.198715877"
Sep 05 19:50:46.282933 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:46.282905    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp"
Sep 05 19:50:46.358375 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:46.358337    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1537c346-b2c8-4e9c-9aa0-6ede348bff59-util\") pod \"1537c346-b2c8-4e9c-9aa0-6ede348bff59\" (UID: \"1537c346-b2c8-4e9c-9aa0-6ede348bff59\") "
Sep 05 19:50:46.358545 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:46.358391    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1537c346-b2c8-4e9c-9aa0-6ede348bff59-bundle\") pod \"1537c346-b2c8-4e9c-9aa0-6ede348bff59\" (UID: \"1537c346-b2c8-4e9c-9aa0-6ede348bff59\") "
Sep 05 19:50:46.358545 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:46.358419    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzr4g\" (UniqueName: \"kubernetes.io/projected/1537c346-b2c8-4e9c-9aa0-6ede348bff59-kube-api-access-tzr4g\") pod \"1537c346-b2c8-4e9c-9aa0-6ede348bff59\" (UID: \"1537c346-b2c8-4e9c-9aa0-6ede348bff59\") "
Sep 05 19:50:46.358740 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:46.358716    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1537c346-b2c8-4e9c-9aa0-6ede348bff59-bundle" (OuterVolumeSpecName: "bundle") pod "1537c346-b2c8-4e9c-9aa0-6ede348bff59" (UID: "1537c346-b2c8-4e9c-9aa0-6ede348bff59"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:50:46.360480 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:46.360452    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1537c346-b2c8-4e9c-9aa0-6ede348bff59-kube-api-access-tzr4g" (OuterVolumeSpecName: "kube-api-access-tzr4g") pod "1537c346-b2c8-4e9c-9aa0-6ede348bff59" (UID: "1537c346-b2c8-4e9c-9aa0-6ede348bff59"). InnerVolumeSpecName "kube-api-access-tzr4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:50:46.362572 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:46.362551    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1537c346-b2c8-4e9c-9aa0-6ede348bff59-util" (OuterVolumeSpecName: "util") pod "1537c346-b2c8-4e9c-9aa0-6ede348bff59" (UID: "1537c346-b2c8-4e9c-9aa0-6ede348bff59"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:50:46.459443 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:46.459370    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1537c346-b2c8-4e9c-9aa0-6ede348bff59-util\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:50:46.459443 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:46.459397    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1537c346-b2c8-4e9c-9aa0-6ede348bff59-bundle\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:50:46.459443 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:46.459406    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzr4g\" (UniqueName: \"kubernetes.io/projected/1537c346-b2c8-4e9c-9aa0-6ede348bff59-kube-api-access-tzr4g\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:50:47.156682 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:47.156654    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp"
Sep 05 19:50:47.156682 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:47.156675    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6mprp" event={"ID":"1537c346-b2c8-4e9c-9aa0-6ede348bff59","Type":"ContainerDied","Data":"cd4bbd54c450ad914b33295de5cceab5d72d7831b919748ef755e863e11b2844"}
Sep 05 19:50:47.157150 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:47.156719    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd4bbd54c450ad914b33295de5cceab5d72d7831b919748ef755e863e11b2844"
Sep 05 19:50:51.099408 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:51.099373    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-hr4jw"]
Sep 05 19:50:51.099767 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:51.099632    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1537c346-b2c8-4e9c-9aa0-6ede348bff59" containerName="extract"
Sep 05 19:50:51.099767 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:51.099642    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1537c346-b2c8-4e9c-9aa0-6ede348bff59" containerName="extract"
Sep 05 19:50:51.099767 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:51.099656    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1537c346-b2c8-4e9c-9aa0-6ede348bff59" containerName="pull"
Sep 05 19:50:51.099767 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:51.099661    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1537c346-b2c8-4e9c-9aa0-6ede348bff59" containerName="pull"
Sep 05 19:50:51.099767 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:51.099674    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1537c346-b2c8-4e9c-9aa0-6ede348bff59" containerName="util"
Sep 05 19:50:51.099767 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:51.099679    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1537c346-b2c8-4e9c-9aa0-6ede348bff59" containerName="util"
Sep 05 19:50:51.099767 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:51.099725    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1537c346-b2c8-4e9c-9aa0-6ede348bff59" containerName="extract"
Sep 05 19:50:51.103272 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:51.103255    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-hr4jw"
Sep 05 19:50:51.105062 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:51.105043    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-8k4lb\""
Sep 05 19:50:51.109326 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:51.109292    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-hr4jw"]
Sep 05 19:50:51.195510 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:51.195476    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-227fg\" (UniqueName: \"kubernetes.io/projected/5d298c1f-9ade-4b62-b56d-f9368aa9cfeb-kube-api-access-227fg\") pod \"cert-manager-55b4d954d5-hr4jw\" (UID: \"5d298c1f-9ade-4b62-b56d-f9368aa9cfeb\") " pod="cert-manager/cert-manager-55b4d954d5-hr4jw"
Sep 05 19:50:51.195510 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:51.195522    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d298c1f-9ade-4b62-b56d-f9368aa9cfeb-bound-sa-token\") pod \"cert-manager-55b4d954d5-hr4jw\" (UID: \"5d298c1f-9ade-4b62-b56d-f9368aa9cfeb\") " pod="cert-manager/cert-manager-55b4d954d5-hr4jw"
Sep 05 19:50:51.296453 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:51.296411    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-227fg\" (UniqueName: \"kubernetes.io/projected/5d298c1f-9ade-4b62-b56d-f9368aa9cfeb-kube-api-access-227fg\") pod \"cert-manager-55b4d954d5-hr4jw\" (UID: \"5d298c1f-9ade-4b62-b56d-f9368aa9cfeb\") " pod="cert-manager/cert-manager-55b4d954d5-hr4jw"
Sep 05 19:50:51.296453 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:51.296457    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d298c1f-9ade-4b62-b56d-f9368aa9cfeb-bound-sa-token\") pod \"cert-manager-55b4d954d5-hr4jw\" (UID: \"5d298c1f-9ade-4b62-b56d-f9368aa9cfeb\") " pod="cert-manager/cert-manager-55b4d954d5-hr4jw"
Sep 05 19:50:51.304157 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:51.304126    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d298c1f-9ade-4b62-b56d-f9368aa9cfeb-bound-sa-token\") pod \"cert-manager-55b4d954d5-hr4jw\" (UID: \"5d298c1f-9ade-4b62-b56d-f9368aa9cfeb\") " pod="cert-manager/cert-manager-55b4d954d5-hr4jw"
Sep 05 19:50:51.304262 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:51.304167    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-227fg\" (UniqueName: \"kubernetes.io/projected/5d298c1f-9ade-4b62-b56d-f9368aa9cfeb-kube-api-access-227fg\") pod \"cert-manager-55b4d954d5-hr4jw\" (UID: \"5d298c1f-9ade-4b62-b56d-f9368aa9cfeb\") " pod="cert-manager/cert-manager-55b4d954d5-hr4jw"
Sep 05 19:50:51.413081 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:51.413001    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-hr4jw"
Sep 05 19:50:51.683308 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:51.683233    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-hr4jw"]
Sep 05 19:50:51.687466 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:50:51.687426    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d298c1f_9ade_4b62_b56d_f9368aa9cfeb.slice/crio-fc451018ec098922a5b80669ef9aa545f99dfc06619a8fcc013587d45ebcc687 WatchSource:0}: Error finding container fc451018ec098922a5b80669ef9aa545f99dfc06619a8fcc013587d45ebcc687: Status 404 returned error can't find the container with id fc451018ec098922a5b80669ef9aa545f99dfc06619a8fcc013587d45ebcc687
Sep 05 19:50:52.178720 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:52.178676    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-hr4jw" event={"ID":"5d298c1f-9ade-4b62-b56d-f9368aa9cfeb","Type":"ContainerStarted","Data":"14bc1d79b044b76c364f3858ecd8381b47874215202cb822866013fc7042a6e3"}
Sep 05 19:50:52.179227 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:52.178725    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-hr4jw" event={"ID":"5d298c1f-9ade-4b62-b56d-f9368aa9cfeb","Type":"ContainerStarted","Data":"fc451018ec098922a5b80669ef9aa545f99dfc06619a8fcc013587d45ebcc687"}
Sep 05 19:50:52.195750 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:50:52.195699    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-hr4jw" podStartSLOduration=1.195679989 podStartE2EDuration="1.195679989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:50:52.194210931 +0000 UTC m=+451.223842837" watchObservedRunningTime="2026-09-05 19:50:52.195679989 +0000 UTC m=+451.225311895"
Sep 05 19:51:00.315415 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:00.315377    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6"]
Sep 05 19:51:00.318837 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:00.318820    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6"
Sep 05 19:51:00.320886 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:00.320865    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vvg94\""
Sep 05 19:51:00.320886 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:00.320880    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 05 19:51:00.321401 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:00.321385    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 05 19:51:00.331710 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:00.331685    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6"]
Sep 05 19:51:00.472095 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:00.472046    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f81a869-3f09-44a7-8ed8-612a66a8a9b3-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6\" (UID: \"0f81a869-3f09-44a7-8ed8-612a66a8a9b3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6"
Sep 05 19:51:00.472267 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:00.472172    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nxvg\" (UniqueName: \"kubernetes.io/projected/0f81a869-3f09-44a7-8ed8-612a66a8a9b3-kube-api-access-9nxvg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6\" (UID: \"0f81a869-3f09-44a7-8ed8-612a66a8a9b3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6"
Sep 05 19:51:00.472267 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:00.472205    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f81a869-3f09-44a7-8ed8-612a66a8a9b3-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6\" (UID: \"0f81a869-3f09-44a7-8ed8-612a66a8a9b3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6"
Sep 05 19:51:00.573409 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:00.573367    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f81a869-3f09-44a7-8ed8-612a66a8a9b3-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6\" (UID: \"0f81a869-3f09-44a7-8ed8-612a66a8a9b3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6"
Sep 05 19:51:00.573559 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:00.573472    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f81a869-3f09-44a7-8ed8-612a66a8a9b3-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6\" (UID: \"0f81a869-3f09-44a7-8ed8-612a66a8a9b3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6"
Sep 05 19:51:00.573559 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:00.573500    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nxvg\" (UniqueName: \"kubernetes.io/projected/0f81a869-3f09-44a7-8ed8-612a66a8a9b3-kube-api-access-9nxvg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6\" (UID: \"0f81a869-3f09-44a7-8ed8-612a66a8a9b3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6"
Sep 05 19:51:00.573836 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:00.573812    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f81a869-3f09-44a7-8ed8-612a66a8a9b3-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6\" (UID: \"0f81a869-3f09-44a7-8ed8-612a66a8a9b3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6"
Sep 05 19:51:00.573875 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:00.573823    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f81a869-3f09-44a7-8ed8-612a66a8a9b3-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6\" (UID: \"0f81a869-3f09-44a7-8ed8-612a66a8a9b3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6"
Sep 05 19:51:00.582421 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:00.582399    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nxvg\" (UniqueName: \"kubernetes.io/projected/0f81a869-3f09-44a7-8ed8-612a66a8a9b3-kube-api-access-9nxvg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6\" (UID: \"0f81a869-3f09-44a7-8ed8-612a66a8a9b3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6"
Sep 05 19:51:00.628062 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:00.628034    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6"
Sep 05 19:51:00.896381 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:00.896300    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6"]
Sep 05 19:51:00.899312 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:51:00.899281    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f81a869_3f09_44a7_8ed8_612a66a8a9b3.slice/crio-085a7eb0a6971df29bdfc854a4286e92f044f07af65b762eec5e2b40351a91b8 WatchSource:0}: Error finding container 085a7eb0a6971df29bdfc854a4286e92f044f07af65b762eec5e2b40351a91b8: Status 404 returned error can't find the container with id 085a7eb0a6971df29bdfc854a4286e92f044f07af65b762eec5e2b40351a91b8
Sep 05 19:51:01.216461 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:01.216357    2583 generic.go:356] "Generic (PLEG): container finished" podID="0f81a869-3f09-44a7-8ed8-612a66a8a9b3" containerID="1e0d073e5cdde5b680eef5a6b42c66523bdfe2a64df390e534b658ede30a690c" exitCode=0
Sep 05 19:51:01.216616 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:01.216447    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6" event={"ID":"0f81a869-3f09-44a7-8ed8-612a66a8a9b3","Type":"ContainerDied","Data":"1e0d073e5cdde5b680eef5a6b42c66523bdfe2a64df390e534b658ede30a690c"}
Sep 05 19:51:01.216616 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:01.216514    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6" event={"ID":"0f81a869-3f09-44a7-8ed8-612a66a8a9b3","Type":"ContainerStarted","Data":"085a7eb0a6971df29bdfc854a4286e92f044f07af65b762eec5e2b40351a91b8"}
Sep 05 19:51:02.222293 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:02.222209    2583 generic.go:356] "Generic (PLEG): container finished" podID="0f81a869-3f09-44a7-8ed8-612a66a8a9b3" containerID="2ee6ca0187800b69db8f63a3e924fa0995fc852c2f9930296201e9b7a5819bbc" exitCode=0
Sep 05 19:51:02.222293 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:02.222274    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6" event={"ID":"0f81a869-3f09-44a7-8ed8-612a66a8a9b3","Type":"ContainerDied","Data":"2ee6ca0187800b69db8f63a3e924fa0995fc852c2f9930296201e9b7a5819bbc"}
Sep 05 19:51:03.228833 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:03.228793    2583 generic.go:356] "Generic (PLEG): container finished" podID="0f81a869-3f09-44a7-8ed8-612a66a8a9b3" containerID="01c24558b2c91a9d7879770a52f91f5f58d2281ea2ee584d98f86cb0573dfffb" exitCode=0
Sep 05 19:51:03.229205 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:03.228868    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6" event={"ID":"0f81a869-3f09-44a7-8ed8-612a66a8a9b3","Type":"ContainerDied","Data":"01c24558b2c91a9d7879770a52f91f5f58d2281ea2ee584d98f86cb0573dfffb"}
Sep 05 19:51:04.363453 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:04.363424    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6"
Sep 05 19:51:04.404714 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:04.404679    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f81a869-3f09-44a7-8ed8-612a66a8a9b3-util\") pod \"0f81a869-3f09-44a7-8ed8-612a66a8a9b3\" (UID: \"0f81a869-3f09-44a7-8ed8-612a66a8a9b3\") "
Sep 05 19:51:04.404877 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:04.404723    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nxvg\" (UniqueName: \"kubernetes.io/projected/0f81a869-3f09-44a7-8ed8-612a66a8a9b3-kube-api-access-9nxvg\") pod \"0f81a869-3f09-44a7-8ed8-612a66a8a9b3\" (UID: \"0f81a869-3f09-44a7-8ed8-612a66a8a9b3\") "
Sep 05 19:51:04.404877 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:04.404750    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f81a869-3f09-44a7-8ed8-612a66a8a9b3-bundle\") pod \"0f81a869-3f09-44a7-8ed8-612a66a8a9b3\" (UID: \"0f81a869-3f09-44a7-8ed8-612a66a8a9b3\") "
Sep 05 19:51:04.405503 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:04.405468    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f81a869-3f09-44a7-8ed8-612a66a8a9b3-bundle" (OuterVolumeSpecName: "bundle") pod "0f81a869-3f09-44a7-8ed8-612a66a8a9b3" (UID: "0f81a869-3f09-44a7-8ed8-612a66a8a9b3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:51:04.406823 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:04.406797    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f81a869-3f09-44a7-8ed8-612a66a8a9b3-kube-api-access-9nxvg" (OuterVolumeSpecName: "kube-api-access-9nxvg") pod "0f81a869-3f09-44a7-8ed8-612a66a8a9b3" (UID: "0f81a869-3f09-44a7-8ed8-612a66a8a9b3"). InnerVolumeSpecName "kube-api-access-9nxvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:51:04.410647 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:04.410505    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f81a869-3f09-44a7-8ed8-612a66a8a9b3-util" (OuterVolumeSpecName: "util") pod "0f81a869-3f09-44a7-8ed8-612a66a8a9b3" (UID: "0f81a869-3f09-44a7-8ed8-612a66a8a9b3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:51:04.505866 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:04.505777    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f81a869-3f09-44a7-8ed8-612a66a8a9b3-util\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:51:04.505866 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:04.505814    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nxvg\" (UniqueName: \"kubernetes.io/projected/0f81a869-3f09-44a7-8ed8-612a66a8a9b3-kube-api-access-9nxvg\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:51:04.505866 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:04.505826    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f81a869-3f09-44a7-8ed8-612a66a8a9b3-bundle\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:51:05.239226 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:05.239193    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6"
Sep 05 19:51:05.239384 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:05.239199    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvmsl6" event={"ID":"0f81a869-3f09-44a7-8ed8-612a66a8a9b3","Type":"ContainerDied","Data":"085a7eb0a6971df29bdfc854a4286e92f044f07af65b762eec5e2b40351a91b8"}
Sep 05 19:51:05.239384 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:05.239311    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="085a7eb0a6971df29bdfc854a4286e92f044f07af65b762eec5e2b40351a91b8"
Sep 05 19:51:10.115993 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:10.115959    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz"]
Sep 05 19:51:10.116398 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:10.116244    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f81a869-3f09-44a7-8ed8-612a66a8a9b3" containerName="util"
Sep 05 19:51:10.116398 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:10.116256    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f81a869-3f09-44a7-8ed8-612a66a8a9b3" containerName="util"
Sep 05 19:51:10.116398 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:10.116267    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f81a869-3f09-44a7-8ed8-612a66a8a9b3" containerName="pull"
Sep 05 19:51:10.116398 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:10.116273    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f81a869-3f09-44a7-8ed8-612a66a8a9b3" containerName="pull"
Sep 05 19:51:10.116398 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:10.116286    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f81a869-3f09-44a7-8ed8-612a66a8a9b3" containerName="extract"
Sep 05 19:51:10.116398 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:10.116291    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f81a869-3f09-44a7-8ed8-612a66a8a9b3" containerName="extract"
Sep 05 19:51:10.116398 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:10.116341    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f81a869-3f09-44a7-8ed8-612a66a8a9b3" containerName="extract"
Sep 05 19:51:10.120415 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:10.120395    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz"
Sep 05 19:51:10.122514 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:10.122478    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 05 19:51:10.122514 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:10.122481    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 05 19:51:10.122677 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:10.122558    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vvg94\""
Sep 05 19:51:10.128859 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:10.128835    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz"]
Sep 05 19:51:10.142860 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:10.142839    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbm6b\" (UniqueName: \"kubernetes.io/projected/3a195f02-e246-476d-9e97-3da92e35902f-kube-api-access-cbm6b\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz\" (UID: \"3a195f02-e246-476d-9e97-3da92e35902f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz"
Sep 05 19:51:10.142964 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:10.142881    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a195f02-e246-476d-9e97-3da92e35902f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz\" (UID: \"3a195f02-e246-476d-9e97-3da92e35902f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz"
Sep 05 19:51:10.142964 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:10.142902    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a195f02-e246-476d-9e97-3da92e35902f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz\" (UID: \"3a195f02-e246-476d-9e97-3da92e35902f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz"
Sep 05 19:51:10.244212 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:10.244159    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a195f02-e246-476d-9e97-3da92e35902f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz\" (UID: \"3a195f02-e246-476d-9e97-3da92e35902f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz"
Sep 05 19:51:10.244405 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:10.244219    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a195f02-e246-476d-9e97-3da92e35902f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz\" (UID: \"3a195f02-e246-476d-9e97-3da92e35902f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz"
Sep 05 19:51:10.244405 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:10.244269    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbm6b\" (UniqueName: \"kubernetes.io/projected/3a195f02-e246-476d-9e97-3da92e35902f-kube-api-access-cbm6b\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz\" (UID: \"3a195f02-e246-476d-9e97-3da92e35902f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz"
Sep 05 19:51:10.244579 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:10.244554    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a195f02-e246-476d-9e97-3da92e35902f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz\" (UID: \"3a195f02-e246-476d-9e97-3da92e35902f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz"
Sep 05 19:51:10.244644 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:10.244600    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a195f02-e246-476d-9e97-3da92e35902f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz\" (UID: \"3a195f02-e246-476d-9e97-3da92e35902f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz"
Sep 05 19:51:10.262063 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:10.262039    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbm6b\" (UniqueName: \"kubernetes.io/projected/3a195f02-e246-476d-9e97-3da92e35902f-kube-api-access-cbm6b\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz\" (UID: \"3a195f02-e246-476d-9e97-3da92e35902f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz"
Sep 05 19:51:10.430014 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:10.429933    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz"
Sep 05 19:51:10.701627 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:10.701577    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz"]
Sep 05 19:51:10.703757 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:51:10.703729    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a195f02_e246_476d_9e97_3da92e35902f.slice/crio-0fe0bed68d596ed5fddd3598a5a988210d9495f69c46066964b75123dd63af4a WatchSource:0}: Error finding container 0fe0bed68d596ed5fddd3598a5a988210d9495f69c46066964b75123dd63af4a: Status 404 returned error can't find the container with id 0fe0bed68d596ed5fddd3598a5a988210d9495f69c46066964b75123dd63af4a
Sep 05 19:51:11.268196 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:11.268168    2583 generic.go:356] "Generic (PLEG): container finished" podID="3a195f02-e246-476d-9e97-3da92e35902f" containerID="74565fffdda905ae3ccfadf6dad3a29b9c1e4dff3426dc95d5fcd04ace23898d" exitCode=0
Sep 05 19:51:11.268509 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:11.268215    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz" event={"ID":"3a195f02-e246-476d-9e97-3da92e35902f","Type":"ContainerDied","Data":"74565fffdda905ae3ccfadf6dad3a29b9c1e4dff3426dc95d5fcd04ace23898d"}
Sep 05 19:51:11.268509 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:11.268248    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz" event={"ID":"3a195f02-e246-476d-9e97-3da92e35902f","Type":"ContainerStarted","Data":"0fe0bed68d596ed5fddd3598a5a988210d9495f69c46066964b75123dd63af4a"}
Sep 05 19:51:11.964839 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:11.964803    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm"]
Sep 05 19:51:11.967928 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:11.967914    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm"
Sep 05 19:51:11.970308 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:11.970279    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 05 19:51:11.970428 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:11.970342    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 05 19:51:11.970428 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:11.970282    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 05 19:51:11.970428 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:11.970282    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ngx5g\""
Sep 05 19:51:11.970428 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:11.970362    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 05 19:51:11.995477 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:11.995435    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm"]
Sep 05 19:51:12.058051 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:12.058023    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4e640a9e-cca6-4e81-8be3-ccfd4b2845fd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-699tm\" (UID: \"4e640a9e-cca6-4e81-8be3-ccfd4b2845fd\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm"
Sep 05 19:51:12.058222 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:12.058076    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4e640a9e-cca6-4e81-8be3-ccfd4b2845fd-webhook-cert\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-699tm\" (UID: \"4e640a9e-cca6-4e81-8be3-ccfd4b2845fd\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm"
Sep 05 19:51:12.058222 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:12.058141    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e640a9e-cca6-4e81-8be3-ccfd4b2845fd-tmp\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-699tm\" (UID: \"4e640a9e-cca6-4e81-8be3-ccfd4b2845fd\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm"
Sep 05 19:51:12.058222 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:12.058176    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4e640a9e-cca6-4e81-8be3-ccfd4b2845fd-manifests\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-699tm\" (UID: \"4e640a9e-cca6-4e81-8be3-ccfd4b2845fd\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm"
Sep 05 19:51:12.058222 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:12.058194    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5m6c\" (UniqueName: \"kubernetes.io/projected/4e640a9e-cca6-4e81-8be3-ccfd4b2845fd-kube-api-access-n5m6c\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-699tm\" (UID: \"4e640a9e-cca6-4e81-8be3-ccfd4b2845fd\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm"
Sep 05 19:51:12.159122 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:12.159072    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4e640a9e-cca6-4e81-8be3-ccfd4b2845fd-webhook-cert\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-699tm\" (UID: \"4e640a9e-cca6-4e81-8be3-ccfd4b2845fd\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm"
Sep 05 19:51:12.159122 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:12.159126    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e640a9e-cca6-4e81-8be3-ccfd4b2845fd-tmp\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-699tm\" (UID: \"4e640a9e-cca6-4e81-8be3-ccfd4b2845fd\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm"
Sep 05 19:51:12.159314 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:12.159146    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4e640a9e-cca6-4e81-8be3-ccfd4b2845fd-manifests\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-699tm\" (UID: \"4e640a9e-cca6-4e81-8be3-ccfd4b2845fd\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm"
Sep 05 19:51:12.159314 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:12.159164    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5m6c\" (UniqueName: \"kubernetes.io/projected/4e640a9e-cca6-4e81-8be3-ccfd4b2845fd-kube-api-access-n5m6c\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-699tm\" (UID: \"4e640a9e-cca6-4e81-8be3-ccfd4b2845fd\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm"
Sep 05 19:51:12.159314 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:12.159195    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4e640a9e-cca6-4e81-8be3-ccfd4b2845fd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-699tm\" (UID: \"4e640a9e-cca6-4e81-8be3-ccfd4b2845fd\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm"
Sep 05 19:51:12.159590 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:12.159563    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4e640a9e-cca6-4e81-8be3-ccfd4b2845fd-manifests\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-699tm\" (UID: \"4e640a9e-cca6-4e81-8be3-ccfd4b2845fd\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm"
Sep 05 19:51:12.161397 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:12.161367    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e640a9e-cca6-4e81-8be3-ccfd4b2845fd-tmp\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-699tm\" (UID: \"4e640a9e-cca6-4e81-8be3-ccfd4b2845fd\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm"
Sep 05 19:51:12.161610 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:12.161591    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4e640a9e-cca6-4e81-8be3-ccfd4b2845fd-webhook-cert\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-699tm\" (UID: \"4e640a9e-cca6-4e81-8be3-ccfd4b2845fd\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm"
Sep 05 19:51:12.161736 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:12.161714    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4e640a9e-cca6-4e81-8be3-ccfd4b2845fd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-699tm\" (UID: \"4e640a9e-cca6-4e81-8be3-ccfd4b2845fd\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm"
Sep 05 19:51:12.169541 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:12.169519    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5m6c\" (UniqueName: \"kubernetes.io/projected/4e640a9e-cca6-4e81-8be3-ccfd4b2845fd-kube-api-access-n5m6c\") pod \"opendatahub-operator-controller-manager-588fc9dcd8-699tm\" (UID: \"4e640a9e-cca6-4e81-8be3-ccfd4b2845fd\") " pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm"
Sep 05 19:51:12.274758 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:12.274732    2583 generic.go:356] "Generic (PLEG): container finished" podID="3a195f02-e246-476d-9e97-3da92e35902f" containerID="31bfa4ff871971f427e7c3444070d2c860e801023b75ad345bfc2bc0fe5180c0" exitCode=0
Sep 05 19:51:12.275057 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:12.274817    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz" event={"ID":"3a195f02-e246-476d-9e97-3da92e35902f","Type":"ContainerDied","Data":"31bfa4ff871971f427e7c3444070d2c860e801023b75ad345bfc2bc0fe5180c0"}
Sep 05 19:51:12.279539 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:12.278472    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm"
Sep 05 19:51:12.558321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:12.558296    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm"]
Sep 05 19:51:12.560851 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:51:12.560825    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e640a9e_cca6_4e81_8be3_ccfd4b2845fd.slice/crio-1a0a89d40a5ce8b290d7508e72e6bed807958a83d5c3ab7758b6f87b2f080741 WatchSource:0}: Error finding container 1a0a89d40a5ce8b290d7508e72e6bed807958a83d5c3ab7758b6f87b2f080741: Status 404 returned error can't find the container with id 1a0a89d40a5ce8b290d7508e72e6bed807958a83d5c3ab7758b6f87b2f080741
Sep 05 19:51:13.283656 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:13.283616    2583 generic.go:356] "Generic (PLEG): container finished" podID="3a195f02-e246-476d-9e97-3da92e35902f" containerID="a0f49300acd1dcbe2550a534cb5211428fc85884cace7b056e824e4c49f1f027" exitCode=0
Sep 05 19:51:13.284093 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:13.283710    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz" event={"ID":"3a195f02-e246-476d-9e97-3da92e35902f","Type":"ContainerDied","Data":"a0f49300acd1dcbe2550a534cb5211428fc85884cace7b056e824e4c49f1f027"}
Sep 05 19:51:13.285558 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:13.285524    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm" event={"ID":"4e640a9e-cca6-4e81-8be3-ccfd4b2845fd","Type":"ContainerStarted","Data":"1a0a89d40a5ce8b290d7508e72e6bed807958a83d5c3ab7758b6f87b2f080741"}
Sep 05 19:51:15.498352 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:15.498328    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz"
Sep 05 19:51:15.586757 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:15.586720    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a195f02-e246-476d-9e97-3da92e35902f-util\") pod \"3a195f02-e246-476d-9e97-3da92e35902f\" (UID: \"3a195f02-e246-476d-9e97-3da92e35902f\") "
Sep 05 19:51:15.586873 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:15.586765    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a195f02-e246-476d-9e97-3da92e35902f-bundle\") pod \"3a195f02-e246-476d-9e97-3da92e35902f\" (UID: \"3a195f02-e246-476d-9e97-3da92e35902f\") "
Sep 05 19:51:15.586873 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:15.586796    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbm6b\" (UniqueName: \"kubernetes.io/projected/3a195f02-e246-476d-9e97-3da92e35902f-kube-api-access-cbm6b\") pod \"3a195f02-e246-476d-9e97-3da92e35902f\" (UID: \"3a195f02-e246-476d-9e97-3da92e35902f\") "
Sep 05 19:51:15.587476 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:15.587448    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a195f02-e246-476d-9e97-3da92e35902f-bundle" (OuterVolumeSpecName: "bundle") pod "3a195f02-e246-476d-9e97-3da92e35902f" (UID: "3a195f02-e246-476d-9e97-3da92e35902f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:51:15.588704 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:15.588681    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a195f02-e246-476d-9e97-3da92e35902f-kube-api-access-cbm6b" (OuterVolumeSpecName: "kube-api-access-cbm6b") pod "3a195f02-e246-476d-9e97-3da92e35902f" (UID: "3a195f02-e246-476d-9e97-3da92e35902f"). InnerVolumeSpecName "kube-api-access-cbm6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:51:15.592077 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:15.592052    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a195f02-e246-476d-9e97-3da92e35902f-util" (OuterVolumeSpecName: "util") pod "3a195f02-e246-476d-9e97-3da92e35902f" (UID: "3a195f02-e246-476d-9e97-3da92e35902f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:51:15.687471 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:15.687440    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a195f02-e246-476d-9e97-3da92e35902f-bundle\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:51:15.687471 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:15.687466    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbm6b\" (UniqueName: \"kubernetes.io/projected/3a195f02-e246-476d-9e97-3da92e35902f-kube-api-access-cbm6b\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:51:15.687471 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:15.687476    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a195f02-e246-476d-9e97-3da92e35902f-util\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:51:16.300584 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:16.300550    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz" event={"ID":"3a195f02-e246-476d-9e97-3da92e35902f","Type":"ContainerDied","Data":"0fe0bed68d596ed5fddd3598a5a988210d9495f69c46066964b75123dd63af4a"}
Sep 05 19:51:16.300584 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:16.300570    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tp8rz"
Sep 05 19:51:16.300778 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:16.300596    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fe0bed68d596ed5fddd3598a5a988210d9495f69c46066964b75123dd63af4a"
Sep 05 19:51:16.302242 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:16.302220    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm" event={"ID":"4e640a9e-cca6-4e81-8be3-ccfd4b2845fd","Type":"ContainerStarted","Data":"8804ccc2cd9975c83c370654eb2d5c67db6bc6655bb2750e0398842c52643257"}
Sep 05 19:51:17.307443 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:17.307406    2583 generic.go:356] "Generic (PLEG): container finished" podID="4e640a9e-cca6-4e81-8be3-ccfd4b2845fd" containerID="8804ccc2cd9975c83c370654eb2d5c67db6bc6655bb2750e0398842c52643257" exitCode=0
Sep 05 19:51:17.307797 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:17.307479    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm" event={"ID":"4e640a9e-cca6-4e81-8be3-ccfd4b2845fd","Type":"ContainerDied","Data":"8804ccc2cd9975c83c370654eb2d5c67db6bc6655bb2750e0398842c52643257"}
Sep 05 19:51:18.314154 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:18.314055    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm" event={"ID":"4e640a9e-cca6-4e81-8be3-ccfd4b2845fd","Type":"ContainerStarted","Data":"ee120c42a503cf9a1218ffb302e08023c2ddeef7e14cd84f654757882598b930"}
Sep 05 19:51:18.314509 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:18.314233    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm"
Sep 05 19:51:18.334887 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:18.334837    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm" podStartSLOduration=2.7387940090000003 podStartE2EDuration="7.33482092s" totalImagesPullingTime="3.529014539s" totalInitContainerRuntime="1.067012372s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:51:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:51:18.33277607 +0000 UTC m=+477.362407974" watchObservedRunningTime="2026-09-05 19:51:18.33482092 +0000 UTC m=+477.364452889"
Sep 05 19:51:29.321656 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:29.321624    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-588fc9dcd8-699tm"
Sep 05 19:51:32.152534 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:32.152502    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-84fbbb6876-94b2x"]
Sep 05 19:51:32.152996 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:32.152927    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a195f02-e246-476d-9e97-3da92e35902f" containerName="pull"
Sep 05 19:51:32.152996 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:32.152975    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a195f02-e246-476d-9e97-3da92e35902f" containerName="pull"
Sep 05 19:51:32.153151 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:32.153007    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a195f02-e246-476d-9e97-3da92e35902f" containerName="util"
Sep 05 19:51:32.153151 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:32.153015    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a195f02-e246-476d-9e97-3da92e35902f" containerName="util"
Sep 05 19:51:32.153151 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:32.153030    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a195f02-e246-476d-9e97-3da92e35902f" containerName="extract"
Sep 05 19:51:32.153151 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:32.153038    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a195f02-e246-476d-9e97-3da92e35902f" containerName="extract"
Sep 05 19:51:32.153151 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:32.153135    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a195f02-e246-476d-9e97-3da92e35902f" containerName="extract"
Sep 05 19:51:32.162793 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:32.162769    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84fbbb6876-94b2x"
Sep 05 19:51:32.165409 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:32.165385    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 05 19:51:32.165409 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:32.165399    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 05 19:51:32.176887 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:32.176860    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84fbbb6876-94b2x"]
Sep 05 19:51:32.216841 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:32.216809    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26be44b8-815a-4081-b02b-6efbb21a0dca-tmp\") pod \"kube-auth-proxy-84fbbb6876-94b2x\" (UID: \"26be44b8-815a-4081-b02b-6efbb21a0dca\") " pod="openshift-ingress/kube-auth-proxy-84fbbb6876-94b2x"
Sep 05 19:51:32.216841 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:32.216846    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgh6g\" (UniqueName: \"kubernetes.io/projected/26be44b8-815a-4081-b02b-6efbb21a0dca-kube-api-access-zgh6g\") pod \"kube-auth-proxy-84fbbb6876-94b2x\" (UID: \"26be44b8-815a-4081-b02b-6efbb21a0dca\") " pod="openshift-ingress/kube-auth-proxy-84fbbb6876-94b2x"
Sep 05 19:51:32.217135 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:32.216944    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26be44b8-815a-4081-b02b-6efbb21a0dca-tls-certs\") pod \"kube-auth-proxy-84fbbb6876-94b2x\" (UID: \"26be44b8-815a-4081-b02b-6efbb21a0dca\") " pod="openshift-ingress/kube-auth-proxy-84fbbb6876-94b2x"
Sep 05 19:51:32.317570 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:32.317542    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26be44b8-815a-4081-b02b-6efbb21a0dca-tmp\") pod \"kube-auth-proxy-84fbbb6876-94b2x\" (UID: \"26be44b8-815a-4081-b02b-6efbb21a0dca\") " pod="openshift-ingress/kube-auth-proxy-84fbbb6876-94b2x"
Sep 05 19:51:32.317730 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:32.317577    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgh6g\" (UniqueName: \"kubernetes.io/projected/26be44b8-815a-4081-b02b-6efbb21a0dca-kube-api-access-zgh6g\") pod \"kube-auth-proxy-84fbbb6876-94b2x\" (UID: \"26be44b8-815a-4081-b02b-6efbb21a0dca\") " pod="openshift-ingress/kube-auth-proxy-84fbbb6876-94b2x"
Sep 05 19:51:32.317730 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:32.317643    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26be44b8-815a-4081-b02b-6efbb21a0dca-tls-certs\") pod \"kube-auth-proxy-84fbbb6876-94b2x\" (UID: \"26be44b8-815a-4081-b02b-6efbb21a0dca\") " pod="openshift-ingress/kube-auth-proxy-84fbbb6876-94b2x"
Sep 05 19:51:32.317817 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:51:32.317765    2583 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 05 19:51:32.317901 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:51:32.317845    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26be44b8-815a-4081-b02b-6efbb21a0dca-tls-certs podName:26be44b8-815a-4081-b02b-6efbb21a0dca nodeName:}" failed. No retries permitted until 2026-09-05 19:51:32.817824444 +0000 UTC m=+491.847456341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/26be44b8-815a-4081-b02b-6efbb21a0dca-tls-certs") pod "kube-auth-proxy-84fbbb6876-94b2x" (UID: "26be44b8-815a-4081-b02b-6efbb21a0dca") : secret "kube-auth-proxy-tls" not found
Sep 05 19:51:32.319899 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:32.319877    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26be44b8-815a-4081-b02b-6efbb21a0dca-tmp\") pod \"kube-auth-proxy-84fbbb6876-94b2x\" (UID: \"26be44b8-815a-4081-b02b-6efbb21a0dca\") " pod="openshift-ingress/kube-auth-proxy-84fbbb6876-94b2x"
Sep 05 19:51:32.337562 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:32.337534    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgh6g\" (UniqueName: \"kubernetes.io/projected/26be44b8-815a-4081-b02b-6efbb21a0dca-kube-api-access-zgh6g\") pod \"kube-auth-proxy-84fbbb6876-94b2x\" (UID: \"26be44b8-815a-4081-b02b-6efbb21a0dca\") " pod="openshift-ingress/kube-auth-proxy-84fbbb6876-94b2x"
Sep 05 19:51:32.821799 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:32.821763    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26be44b8-815a-4081-b02b-6efbb21a0dca-tls-certs\") pod \"kube-auth-proxy-84fbbb6876-94b2x\" (UID: \"26be44b8-815a-4081-b02b-6efbb21a0dca\") " pod="openshift-ingress/kube-auth-proxy-84fbbb6876-94b2x"
Sep 05 19:51:32.824085 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:32.824063    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26be44b8-815a-4081-b02b-6efbb21a0dca-tls-certs\") pod \"kube-auth-proxy-84fbbb6876-94b2x\" (UID: \"26be44b8-815a-4081-b02b-6efbb21a0dca\") " pod="openshift-ingress/kube-auth-proxy-84fbbb6876-94b2x"
Sep 05 19:51:33.072759 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.072677    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84fbbb6876-94b2x"
Sep 05 19:51:33.366720 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.366669    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84fbbb6876-94b2x"]
Sep 05 19:51:33.368751 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:51:33.368714    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26be44b8_815a_4081_b02b_6efbb21a0dca.slice/crio-91b8074072a119303515625b3e735dcd7a0bf8f6b64e82ddb0b03d0ae1754160 WatchSource:0}: Error finding container 91b8074072a119303515625b3e735dcd7a0bf8f6b64e82ddb0b03d0ae1754160: Status 404 returned error can't find the container with id 91b8074072a119303515625b3e735dcd7a0bf8f6b64e82ddb0b03d0ae1754160
Sep 05 19:51:33.380569 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.380545    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84fbbb6876-94b2x" event={"ID":"26be44b8-815a-4081-b02b-6efbb21a0dca","Type":"ContainerStarted","Data":"91b8074072a119303515625b3e735dcd7a0bf8f6b64e82ddb0b03d0ae1754160"}
Sep 05 19:51:33.740936 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.740856    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"]
Sep 05 19:51:33.745791 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.745768    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:33.747959 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.747934    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 05 19:51:33.748138 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.747958    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 05 19:51:33.748138 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.748040    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 05 19:51:33.748332 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.748313    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 05 19:51:33.748750 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.748736    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-pxwq9\""
Sep 05 19:51:33.761381 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.761360    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"]
Sep 05 19:51:33.828704 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.828663    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac95d07b-d8a6-48ac-9d84-a065844c3012-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-7vh9j\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:33.828865 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.828734    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac95d07b-d8a6-48ac-9d84-a065844c3012-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-7vh9j\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:33.828865 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.828789    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac95d07b-d8a6-48ac-9d84-a065844c3012-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-7vh9j\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:33.828940 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.828871    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac95d07b-d8a6-48ac-9d84-a065844c3012-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-7vh9j\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:33.828940 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.828915    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k69v\" (UniqueName: \"kubernetes.io/projected/ac95d07b-d8a6-48ac-9d84-a065844c3012-kube-api-access-2k69v\") pod \"istiod-openshift-gateway-7b7995d5cd-7vh9j\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:33.829007 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.828963    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac95d07b-d8a6-48ac-9d84-a065844c3012-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-7vh9j\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:33.829007 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.828992    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac95d07b-d8a6-48ac-9d84-a065844c3012-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-7vh9j\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:33.929651 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.929616    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac95d07b-d8a6-48ac-9d84-a065844c3012-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-7vh9j\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:33.929651 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.929655    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac95d07b-d8a6-48ac-9d84-a065844c3012-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-7vh9j\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:33.929857 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.929789    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac95d07b-d8a6-48ac-9d84-a065844c3012-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-7vh9j\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:33.929857 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.929842    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k69v\" (UniqueName: \"kubernetes.io/projected/ac95d07b-d8a6-48ac-9d84-a065844c3012-kube-api-access-2k69v\") pod \"istiod-openshift-gateway-7b7995d5cd-7vh9j\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:33.929941 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.929874    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac95d07b-d8a6-48ac-9d84-a065844c3012-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-7vh9j\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:33.929941 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.929906    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac95d07b-d8a6-48ac-9d84-a065844c3012-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-7vh9j\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:33.930033 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.929961    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac95d07b-d8a6-48ac-9d84-a065844c3012-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-7vh9j\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:33.930466 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.930437    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac95d07b-d8a6-48ac-9d84-a065844c3012-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-7vh9j\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:33.932377 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.932318    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac95d07b-d8a6-48ac-9d84-a065844c3012-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-7vh9j\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:33.932487 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.932377    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac95d07b-d8a6-48ac-9d84-a065844c3012-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-7vh9j\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:33.932487 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.932410    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac95d07b-d8a6-48ac-9d84-a065844c3012-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-7vh9j\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:33.932587 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.932569    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac95d07b-d8a6-48ac-9d84-a065844c3012-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-7vh9j\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:33.938321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.938288    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac95d07b-d8a6-48ac-9d84-a065844c3012-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-7vh9j\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:33.938877 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:33.938852    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k69v\" (UniqueName: \"kubernetes.io/projected/ac95d07b-d8a6-48ac-9d84-a065844c3012-kube-api-access-2k69v\") pod \"istiod-openshift-gateway-7b7995d5cd-7vh9j\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:34.055425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:34.055331    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:34.348438 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:34.348406    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"]
Sep 05 19:51:34.350854 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:51:34.350819    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac95d07b_d8a6_48ac_9d84_a065844c3012.slice/crio-7458266c4e5398dc68c39dfd60349e78de82f04ae0ade80495ef3934df877472 WatchSource:0}: Error finding container 7458266c4e5398dc68c39dfd60349e78de82f04ae0ade80495ef3934df877472: Status 404 returned error can't find the container with id 7458266c4e5398dc68c39dfd60349e78de82f04ae0ade80495ef3934df877472
Sep 05 19:51:34.388876 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:34.388843    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j" event={"ID":"ac95d07b-d8a6-48ac-9d84-a065844c3012","Type":"ContainerStarted","Data":"7458266c4e5398dc68c39dfd60349e78de82f04ae0ade80495ef3934df877472"}
Sep 05 19:51:36.400985 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:36.400947    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84fbbb6876-94b2x" event={"ID":"26be44b8-815a-4081-b02b-6efbb21a0dca","Type":"ContainerStarted","Data":"110a20028a2bc5cb66455f81328fe90f3f49dd0e4ea223e3e2248d297b72ee37"}
Sep 05 19:51:36.418365 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:36.418305    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-84fbbb6876-94b2x" podStartSLOduration=1.7539991289999999 podStartE2EDuration="4.418284454s" totalImagesPullingTime="2.664285325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:51:36.416798831 +0000 UTC m=+495.446430748" watchObservedRunningTime="2026-09-05 19:51:36.418284454 +0000 UTC m=+495.447916360"
Sep 05 19:51:37.790674 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:37.790626    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:51:37.791000 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:37.790712    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:51:38.412479 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.412426    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j" event={"ID":"ac95d07b-d8a6-48ac-9d84-a065844c3012","Type":"ContainerStarted","Data":"7148f095932526aebfaad5132365960c48fd978efa51c794766701866b99dd1c"}
Sep 05 19:51:38.412659 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.412594    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:38.414482 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.414456    2583 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-7vh9j container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 05 19:51:38.414605 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.414551    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j" podUID="ac95d07b-d8a6-48ac-9d84-a065844c3012" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 19:51:38.433225 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.433172    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j" podStartSLOduration=1.9963176379999998 podStartE2EDuration="5.433157908s" totalImagesPullingTime="3.43684027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:51:38.432122922 +0000 UTC m=+497.461754832" watchObservedRunningTime="2026-09-05 19:51:38.433157908 +0000 UTC m=+497.462789810"
Sep 05 19:51:38.489951 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.489916    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"]
Sep 05 19:51:38.493331 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.493302    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.495545 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.495518    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-kv46h\""
Sep 05 19:51:38.507612 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.505065    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"]
Sep 05 19:51:38.575607 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.575572    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.575607 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.575614    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.575802 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.575674    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.575802 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.575749    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.575802 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.575770    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.575941 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.575803    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.575941 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.575829    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.575941 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.575896    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.576150 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.575951    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjq6f\" (UniqueName: \"kubernetes.io/projected/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-kube-api-access-kjq6f\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.676437 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.676360    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjq6f\" (UniqueName: \"kubernetes.io/projected/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-kube-api-access-kjq6f\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.676579 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.676433    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.676579 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.676463    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.676579 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.676499    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.676579 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.676530    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.676579 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.676555    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.676786 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.676580    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.676786 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.676609    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.676786 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.676637    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.676957 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.676930    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.677099 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.677077    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.677193 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.677167    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.677250 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.677200    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.677368 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.677347    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.678980 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.678957    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.679104 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.679087    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.688857 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.688831    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.693048 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.693025    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjq6f\" (UniqueName: \"kubernetes.io/projected/bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38-kube-api-access-kjq6f\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2\" (UID: \"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:38.804042 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:38.804007    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:39.076656 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:39.076624    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"]
Sep 05 19:51:39.079693 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:51:39.079665    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc0e00bd_9a7e_45a5_93d7_2fd24fad5e38.slice/crio-462e474ca3885b4a12d6c61c6c6c04eb8c510fa76045161e5f29d663baaaa552 WatchSource:0}: Error finding container 462e474ca3885b4a12d6c61c6c6c04eb8c510fa76045161e5f29d663baaaa552: Status 404 returned error can't find the container with id 462e474ca3885b4a12d6c61c6c6c04eb8c510fa76045161e5f29d663baaaa552
Sep 05 19:51:39.417732 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:39.417640    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2" event={"ID":"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38","Type":"ContainerStarted","Data":"462e474ca3885b4a12d6c61c6c6c04eb8c510fa76045161e5f29d663baaaa552"}
Sep 05 19:51:39.418499 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:39.418474    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:41.789017 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:41.788980    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:51:41.789292 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:41.789052    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:51:41.789292 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:41.789079    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:51:42.437938 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:42.437901    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2" event={"ID":"bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38","Type":"ContainerStarted","Data":"f31c0202f71b670604e8277a9a7542bd7d1786ba264cbc7d29c45613bc2a3dcb"}
Sep 05 19:51:42.459031 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:42.458984    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2" podStartSLOduration=1.751890771 podStartE2EDuration="4.45896888s" totalImagesPullingTime="2.707078109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:51:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:51:42.456196436 +0000 UTC m=+501.485828343" watchObservedRunningTime="2026-09-05 19:51:42.45896888 +0000 UTC m=+501.488600785"
Sep 05 19:51:42.805135 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:42.805034    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:42.811505 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:42.811479    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:43.442071 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:43.442038    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:43.442917 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:43.442895    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2"
Sep 05 19:51:53.538136 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:53.538072    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-f5chv"]
Sep 05 19:51:53.544971 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:53.544946    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-f5chv"
Sep 05 19:51:53.547533 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:53.547511    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 05 19:51:53.547640 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:53.547619    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wwp2m\""
Sep 05 19:51:53.554233 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:53.554212    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-f5chv"]
Sep 05 19:51:53.559284 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:53.559258    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-627s6"]
Sep 05 19:51:53.564582 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:53.564561    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-627s6"
Sep 05 19:51:53.566841 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:53.566820    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-gskk6\""
Sep 05 19:51:53.566939 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:53.566894    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 05 19:51:53.576058 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:53.576038    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-627s6"]
Sep 05 19:51:53.704403 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:53.704362    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nzqd\" (UniqueName: \"kubernetes.io/projected/31640114-9c04-4863-bd1f-da72c3101a2f-kube-api-access-9nzqd\") pod \"llmisvc-controller-manager-777c6bd7d5-627s6\" (UID: \"31640114-9c04-4863-bd1f-da72c3101a2f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-627s6"
Sep 05 19:51:53.704601 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:53.704424    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vgbv\" (UniqueName: \"kubernetes.io/projected/0efde6da-00a5-46a8-a034-eb18da89c10a-kube-api-access-2vgbv\") pod \"kserve-controller-manager-d6f4b65b8-f5chv\" (UID: \"0efde6da-00a5-46a8-a034-eb18da89c10a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-f5chv"
Sep 05 19:51:53.704601 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:53.704450    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31640114-9c04-4863-bd1f-da72c3101a2f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-627s6\" (UID: \"31640114-9c04-4863-bd1f-da72c3101a2f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-627s6"
Sep 05 19:51:53.704601 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:53.704529    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0efde6da-00a5-46a8-a034-eb18da89c10a-cert\") pod \"kserve-controller-manager-d6f4b65b8-f5chv\" (UID: \"0efde6da-00a5-46a8-a034-eb18da89c10a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-f5chv"
Sep 05 19:51:53.805125 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:53.805015    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0efde6da-00a5-46a8-a034-eb18da89c10a-cert\") pod \"kserve-controller-manager-d6f4b65b8-f5chv\" (UID: \"0efde6da-00a5-46a8-a034-eb18da89c10a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-f5chv"
Sep 05 19:51:53.805125 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:53.805098    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nzqd\" (UniqueName: \"kubernetes.io/projected/31640114-9c04-4863-bd1f-da72c3101a2f-kube-api-access-9nzqd\") pod \"llmisvc-controller-manager-777c6bd7d5-627s6\" (UID: \"31640114-9c04-4863-bd1f-da72c3101a2f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-627s6"
Sep 05 19:51:53.805296 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:51:53.805194    2583 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 05 19:51:53.805296 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:51:53.805264    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0efde6da-00a5-46a8-a034-eb18da89c10a-cert podName:0efde6da-00a5-46a8-a034-eb18da89c10a nodeName:}" failed. No retries permitted until 2026-09-05 19:51:54.305245153 +0000 UTC m=+513.334877047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0efde6da-00a5-46a8-a034-eb18da89c10a-cert") pod "kserve-controller-manager-d6f4b65b8-f5chv" (UID: "0efde6da-00a5-46a8-a034-eb18da89c10a") : secret "kserve-webhook-server-cert" not found
Sep 05 19:51:53.805296 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:53.805279    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vgbv\" (UniqueName: \"kubernetes.io/projected/0efde6da-00a5-46a8-a034-eb18da89c10a-kube-api-access-2vgbv\") pod \"kserve-controller-manager-d6f4b65b8-f5chv\" (UID: \"0efde6da-00a5-46a8-a034-eb18da89c10a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-f5chv"
Sep 05 19:51:53.805424 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:53.805308    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31640114-9c04-4863-bd1f-da72c3101a2f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-627s6\" (UID: \"31640114-9c04-4863-bd1f-da72c3101a2f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-627s6"
Sep 05 19:51:53.805424 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:51:53.805398    2583 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 05 19:51:53.805489 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:51:53.805433    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31640114-9c04-4863-bd1f-da72c3101a2f-cert podName:31640114-9c04-4863-bd1f-da72c3101a2f nodeName:}" failed. No retries permitted until 2026-09-05 19:51:54.30542143 +0000 UTC m=+513.335053321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/31640114-9c04-4863-bd1f-da72c3101a2f-cert") pod "llmisvc-controller-manager-777c6bd7d5-627s6" (UID: "31640114-9c04-4863-bd1f-da72c3101a2f") : secret "llmisvc-webhook-server-cert" not found
Sep 05 19:51:53.828302 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:53.828268    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vgbv\" (UniqueName: \"kubernetes.io/projected/0efde6da-00a5-46a8-a034-eb18da89c10a-kube-api-access-2vgbv\") pod \"kserve-controller-manager-d6f4b65b8-f5chv\" (UID: \"0efde6da-00a5-46a8-a034-eb18da89c10a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-f5chv"
Sep 05 19:51:53.831243 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:53.831222    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nzqd\" (UniqueName: \"kubernetes.io/projected/31640114-9c04-4863-bd1f-da72c3101a2f-kube-api-access-9nzqd\") pod \"llmisvc-controller-manager-777c6bd7d5-627s6\" (UID: \"31640114-9c04-4863-bd1f-da72c3101a2f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-627s6"
Sep 05 19:51:54.248250 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.248214    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"]
Sep 05 19:51:54.248462 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.248423    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j" podUID="ac95d07b-d8a6-48ac-9d84-a065844c3012" containerName="discovery" containerID="cri-o://7148f095932526aebfaad5132365960c48fd978efa51c794766701866b99dd1c" gracePeriod=30
Sep 05 19:51:54.310081 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.310045    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0efde6da-00a5-46a8-a034-eb18da89c10a-cert\") pod \"kserve-controller-manager-d6f4b65b8-f5chv\" (UID: \"0efde6da-00a5-46a8-a034-eb18da89c10a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-f5chv"
Sep 05 19:51:54.310274 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.310179    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31640114-9c04-4863-bd1f-da72c3101a2f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-627s6\" (UID: \"31640114-9c04-4863-bd1f-da72c3101a2f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-627s6"
Sep 05 19:51:54.312441 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.312418    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/31640114-9c04-4863-bd1f-da72c3101a2f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-627s6\" (UID: \"31640114-9c04-4863-bd1f-da72c3101a2f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-627s6"
Sep 05 19:51:54.312535 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.312482    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0efde6da-00a5-46a8-a034-eb18da89c10a-cert\") pod \"kserve-controller-manager-d6f4b65b8-f5chv\" (UID: \"0efde6da-00a5-46a8-a034-eb18da89c10a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-f5chv"
Sep 05 19:51:54.455980 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.455951    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-f5chv"
Sep 05 19:51:54.474914 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.474887    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-627s6"
Sep 05 19:51:54.488453 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.485178    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:54.493134 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.493092    2583 generic.go:356] "Generic (PLEG): container finished" podID="ac95d07b-d8a6-48ac-9d84-a065844c3012" containerID="7148f095932526aebfaad5132365960c48fd978efa51c794766701866b99dd1c" exitCode=0
Sep 05 19:51:54.493244 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.493157    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j" event={"ID":"ac95d07b-d8a6-48ac-9d84-a065844c3012","Type":"ContainerDied","Data":"7148f095932526aebfaad5132365960c48fd978efa51c794766701866b99dd1c"}
Sep 05 19:51:54.493244 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.493229    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j" event={"ID":"ac95d07b-d8a6-48ac-9d84-a065844c3012","Type":"ContainerDied","Data":"7458266c4e5398dc68c39dfd60349e78de82f04ae0ade80495ef3934df877472"}
Sep 05 19:51:54.493320 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.493251    2583 scope.go:117] "RemoveContainer" containerID="7148f095932526aebfaad5132365960c48fd978efa51c794766701866b99dd1c"
Sep 05 19:51:54.493320 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.493168    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"
Sep 05 19:51:54.506316 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.506259    2583 scope.go:117] "RemoveContainer" containerID="7148f095932526aebfaad5132365960c48fd978efa51c794766701866b99dd1c"
Sep 05 19:51:54.506838 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:51:54.506732    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7148f095932526aebfaad5132365960c48fd978efa51c794766701866b99dd1c\": container with ID starting with 7148f095932526aebfaad5132365960c48fd978efa51c794766701866b99dd1c not found: ID does not exist" containerID="7148f095932526aebfaad5132365960c48fd978efa51c794766701866b99dd1c"
Sep 05 19:51:54.506838 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.506766    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7148f095932526aebfaad5132365960c48fd978efa51c794766701866b99dd1c"} err="failed to get container status \"7148f095932526aebfaad5132365960c48fd978efa51c794766701866b99dd1c\": rpc error: code = NotFound desc = could not find container \"7148f095932526aebfaad5132365960c48fd978efa51c794766701866b99dd1c\": container with ID starting with 7148f095932526aebfaad5132365960c48fd978efa51c794766701866b99dd1c not found: ID does not exist"
Sep 05 19:51:54.613548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.612762    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac95d07b-d8a6-48ac-9d84-a065844c3012-local-certs\") pod \"ac95d07b-d8a6-48ac-9d84-a065844c3012\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") "
Sep 05 19:51:54.613548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.612816    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac95d07b-d8a6-48ac-9d84-a065844c3012-istio-csr-dns-cert\") pod \"ac95d07b-d8a6-48ac-9d84-a065844c3012\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") "
Sep 05 19:51:54.613548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.612839    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac95d07b-d8a6-48ac-9d84-a065844c3012-istio-csr-ca-configmap\") pod \"ac95d07b-d8a6-48ac-9d84-a065844c3012\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") "
Sep 05 19:51:54.613548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.612862    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac95d07b-d8a6-48ac-9d84-a065844c3012-istio-token\") pod \"ac95d07b-d8a6-48ac-9d84-a065844c3012\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") "
Sep 05 19:51:54.613548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.612906    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k69v\" (UniqueName: \"kubernetes.io/projected/ac95d07b-d8a6-48ac-9d84-a065844c3012-kube-api-access-2k69v\") pod \"ac95d07b-d8a6-48ac-9d84-a065844c3012\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") "
Sep 05 19:51:54.613548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.612947    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac95d07b-d8a6-48ac-9d84-a065844c3012-cacerts\") pod \"ac95d07b-d8a6-48ac-9d84-a065844c3012\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") "
Sep 05 19:51:54.613548 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.613013    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac95d07b-d8a6-48ac-9d84-a065844c3012-istio-kubeconfig\") pod \"ac95d07b-d8a6-48ac-9d84-a065844c3012\" (UID: \"ac95d07b-d8a6-48ac-9d84-a065844c3012\") "
Sep 05 19:51:54.615857 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.615803    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac95d07b-d8a6-48ac-9d84-a065844c3012-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "ac95d07b-d8a6-48ac-9d84-a065844c3012" (UID: "ac95d07b-d8a6-48ac-9d84-a065844c3012"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 19:51:54.618575 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.618516    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac95d07b-d8a6-48ac-9d84-a065844c3012-cacerts" (OuterVolumeSpecName: "cacerts") pod "ac95d07b-d8a6-48ac-9d84-a065844c3012" (UID: "ac95d07b-d8a6-48ac-9d84-a065844c3012"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 19:51:54.618575 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.618549    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac95d07b-d8a6-48ac-9d84-a065844c3012-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "ac95d07b-d8a6-48ac-9d84-a065844c3012" (UID: "ac95d07b-d8a6-48ac-9d84-a065844c3012"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 19:51:54.618740 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.618649    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac95d07b-d8a6-48ac-9d84-a065844c3012-local-certs" (OuterVolumeSpecName: "local-certs") pod "ac95d07b-d8a6-48ac-9d84-a065844c3012" (UID: "ac95d07b-d8a6-48ac-9d84-a065844c3012"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:51:54.618807 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.618781    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac95d07b-d8a6-48ac-9d84-a065844c3012-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "ac95d07b-d8a6-48ac-9d84-a065844c3012" (UID: "ac95d07b-d8a6-48ac-9d84-a065844c3012"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 19:51:54.619015 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.618989    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac95d07b-d8a6-48ac-9d84-a065844c3012-istio-token" (OuterVolumeSpecName: "istio-token") pod "ac95d07b-d8a6-48ac-9d84-a065844c3012" (UID: "ac95d07b-d8a6-48ac-9d84-a065844c3012"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:51:54.619850 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.619826    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac95d07b-d8a6-48ac-9d84-a065844c3012-kube-api-access-2k69v" (OuterVolumeSpecName: "kube-api-access-2k69v") pod "ac95d07b-d8a6-48ac-9d84-a065844c3012" (UID: "ac95d07b-d8a6-48ac-9d84-a065844c3012"). InnerVolumeSpecName "kube-api-access-2k69v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:51:54.714146 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.714000    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k69v\" (UniqueName: \"kubernetes.io/projected/ac95d07b-d8a6-48ac-9d84-a065844c3012-kube-api-access-2k69v\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:51:54.714146 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.714034    2583 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac95d07b-d8a6-48ac-9d84-a065844c3012-cacerts\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:51:54.714146 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.714051    2583 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac95d07b-d8a6-48ac-9d84-a065844c3012-istio-kubeconfig\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:51:54.714146 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.714064    2583 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac95d07b-d8a6-48ac-9d84-a065844c3012-local-certs\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:51:54.714146 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.714077    2583 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac95d07b-d8a6-48ac-9d84-a065844c3012-istio-csr-dns-cert\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:51:54.714146 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.714090    2583 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac95d07b-d8a6-48ac-9d84-a065844c3012-istio-csr-ca-configmap\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:51:54.714146 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.714103    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac95d07b-d8a6-48ac-9d84-a065844c3012-istio-token\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:51:54.749379 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.749345    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-f5chv"]
Sep 05 19:51:54.750468 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:51:54.750436    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0efde6da_00a5_46a8_a034_eb18da89c10a.slice/crio-8245e5223a356b66c06169126d28dd142ce5869150d452400879de5e2559391c WatchSource:0}: Error finding container 8245e5223a356b66c06169126d28dd142ce5869150d452400879de5e2559391c: Status 404 returned error can't find the container with id 8245e5223a356b66c06169126d28dd142ce5869150d452400879de5e2559391c
Sep 05 19:51:54.780084 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.780048    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-627s6"]
Sep 05 19:51:54.781775 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:51:54.781730    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod31640114_9c04_4863_bd1f_da72c3101a2f.slice/crio-5e1e5b809dad2c6d2e7a42c4e43de11c2dbd87e64c7aa871f980bf246465eadf WatchSource:0}: Error finding container 5e1e5b809dad2c6d2e7a42c4e43de11c2dbd87e64c7aa871f980bf246465eadf: Status 404 returned error can't find the container with id 5e1e5b809dad2c6d2e7a42c4e43de11c2dbd87e64c7aa871f980bf246465eadf
Sep 05 19:51:54.830635 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.830602    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"]
Sep 05 19:51:54.855198 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:54.855151    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j"]
Sep 05 19:51:55.436476 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:55.436435    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac95d07b-d8a6-48ac-9d84-a065844c3012" path="/var/lib/kubelet/pods/ac95d07b-d8a6-48ac-9d84-a065844c3012/volumes"
Sep 05 19:51:55.500130 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:55.500072    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-f5chv" event={"ID":"0efde6da-00a5-46a8-a034-eb18da89c10a","Type":"ContainerStarted","Data":"8245e5223a356b66c06169126d28dd142ce5869150d452400879de5e2559391c"}
Sep 05 19:51:55.502549 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:55.502519    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-627s6" event={"ID":"31640114-9c04-4863-bd1f-da72c3101a2f","Type":"ContainerStarted","Data":"5e1e5b809dad2c6d2e7a42c4e43de11c2dbd87e64c7aa871f980bf246465eadf"}
Sep 05 19:51:58.518166 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:58.518129    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-f5chv" event={"ID":"0efde6da-00a5-46a8-a034-eb18da89c10a","Type":"ContainerStarted","Data":"28e2b595cd6ae9dc26627183fee5eeef379fb7baa9a825e04d55520b8a80e1cf"}
Sep 05 19:51:58.518588 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:58.518193    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-f5chv"
Sep 05 19:51:58.519660 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:58.519633    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-627s6" event={"ID":"31640114-9c04-4863-bd1f-da72c3101a2f","Type":"ContainerStarted","Data":"aed713ccbee959da645f26c55d92a894a773a4659d0b8fd4ffdc87c6927c143d"}
Sep 05 19:51:58.519762 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:58.519688    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-627s6"
Sep 05 19:51:58.536014 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:58.535971    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-f5chv" podStartSLOduration=2.815592632 podStartE2EDuration="5.535956567s" totalImagesPullingTime="2.720363935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:51:58.534263805 +0000 UTC m=+517.563895710" watchObservedRunningTime="2026-09-05 19:51:58.535956567 +0000 UTC m=+517.565588470"
Sep 05 19:51:58.550595 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:58.550547    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-627s6" podStartSLOduration=2.809046618 podStartE2EDuration="5.550528317s" totalImagesPullingTime="2.741481699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:51:58.549826382 +0000 UTC m=+517.579458289" watchObservedRunningTime="2026-09-05 19:51:58.550528317 +0000 UTC m=+517.580160223"
Sep 05 19:51:59.418756 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:59.418712    2583 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-7vh9j container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.26:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 05 19:51:59.418925 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:51:59.418784    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7vh9j" podUID="ac95d07b-d8a6-48ac-9d84-a065844c3012" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.26:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 05 19:52:29.527124 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:29.527089    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-627s6"
Sep 05 19:52:29.530531 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:29.530506    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-f5chv"
Sep 05 19:52:47.331048 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.331008    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc"]
Sep 05 19:52:47.331425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.331349    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac95d07b-d8a6-48ac-9d84-a065844c3012" containerName="discovery"
Sep 05 19:52:47.331425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.331360    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac95d07b-d8a6-48ac-9d84-a065844c3012" containerName="discovery"
Sep 05 19:52:47.331425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.331416    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac95d07b-d8a6-48ac-9d84-a065844c3012" containerName="discovery"
Sep 05 19:52:47.334194 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.334176    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc"
Sep 05 19:52:47.336628 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.336608    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 05 19:52:47.336716 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.336649    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vvg94\""
Sep 05 19:52:47.337224 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.337208    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 05 19:52:47.343994 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.343968    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc"]
Sep 05 19:52:47.391477 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.391444    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr"]
Sep 05 19:52:47.394629 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.394610    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr"
Sep 05 19:52:47.403858 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.403834    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr"]
Sep 05 19:52:47.454853 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.454815    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/741f53bb-bad1-4881-a680-7fc4b1a20fd0-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc\" (UID: \"741f53bb-bad1-4881-a680-7fc4b1a20fd0\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc"
Sep 05 19:52:47.454853 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.454854    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nwl5\" (UniqueName: \"kubernetes.io/projected/741f53bb-bad1-4881-a680-7fc4b1a20fd0-kube-api-access-6nwl5\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc\" (UID: \"741f53bb-bad1-4881-a680-7fc4b1a20fd0\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc"
Sep 05 19:52:47.455045 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.454930    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/741f53bb-bad1-4881-a680-7fc4b1a20fd0-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc\" (UID: \"741f53bb-bad1-4881-a680-7fc4b1a20fd0\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc"
Sep 05 19:52:47.483198 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.483153    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr"]
Sep 05 19:52:47.486428 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.486407    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr"
Sep 05 19:52:47.495544 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.495511    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr"]
Sep 05 19:52:47.555524 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.555487    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbc01914-1b88-49e5-a2f9-43625bd7a214-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr\" (UID: \"dbc01914-1b88-49e5-a2f9-43625bd7a214\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr"
Sep 05 19:52:47.555524 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.555525    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2srm\" (UniqueName: \"kubernetes.io/projected/dbc01914-1b88-49e5-a2f9-43625bd7a214-kube-api-access-b2srm\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr\" (UID: \"dbc01914-1b88-49e5-a2f9-43625bd7a214\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr"
Sep 05 19:52:47.555733 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.555596    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/741f53bb-bad1-4881-a680-7fc4b1a20fd0-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc\" (UID: \"741f53bb-bad1-4881-a680-7fc4b1a20fd0\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc"
Sep 05 19:52:47.555733 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.555632    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nwl5\" (UniqueName: \"kubernetes.io/projected/741f53bb-bad1-4881-a680-7fc4b1a20fd0-kube-api-access-6nwl5\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc\" (UID: \"741f53bb-bad1-4881-a680-7fc4b1a20fd0\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc"
Sep 05 19:52:47.555733 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.555690    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr\" (UID: \"9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr"
Sep 05 19:52:47.555733 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.555716    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr\" (UID: \"9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr"
Sep 05 19:52:47.555891 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.555735    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvhbm\" (UniqueName: \"kubernetes.io/projected/9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99-kube-api-access-vvhbm\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr\" (UID: \"9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr"
Sep 05 19:52:47.555891 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.555766    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/741f53bb-bad1-4881-a680-7fc4b1a20fd0-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc\" (UID: \"741f53bb-bad1-4881-a680-7fc4b1a20fd0\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc"
Sep 05 19:52:47.555891 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.555814    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbc01914-1b88-49e5-a2f9-43625bd7a214-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr\" (UID: \"dbc01914-1b88-49e5-a2f9-43625bd7a214\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr"
Sep 05 19:52:47.556004 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.555991    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/741f53bb-bad1-4881-a680-7fc4b1a20fd0-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc\" (UID: \"741f53bb-bad1-4881-a680-7fc4b1a20fd0\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc"
Sep 05 19:52:47.556041 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.556012    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/741f53bb-bad1-4881-a680-7fc4b1a20fd0-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc\" (UID: \"741f53bb-bad1-4881-a680-7fc4b1a20fd0\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc"
Sep 05 19:52:47.563867 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.563837    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nwl5\" (UniqueName: \"kubernetes.io/projected/741f53bb-bad1-4881-a680-7fc4b1a20fd0-kube-api-access-6nwl5\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc\" (UID: \"741f53bb-bad1-4881-a680-7fc4b1a20fd0\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc"
Sep 05 19:52:47.583917 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.583846    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx"]
Sep 05 19:52:47.587176 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.587160    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx"
Sep 05 19:52:47.596358 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.596330    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx"]
Sep 05 19:52:47.643874 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.643831    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc"
Sep 05 19:52:47.656787 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.656755    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cktls\" (UniqueName: \"kubernetes.io/projected/c09d415c-bc67-4c32-82f5-ec6490dd3d7b-kube-api-access-cktls\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx\" (UID: \"c09d415c-bc67-4c32-82f5-ec6490dd3d7b\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx"
Sep 05 19:52:47.656927 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.656815    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr\" (UID: \"9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr"
Sep 05 19:52:47.657030 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.656926    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr\" (UID: \"9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr"
Sep 05 19:52:47.657030 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.656985    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvhbm\" (UniqueName: \"kubernetes.io/projected/9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99-kube-api-access-vvhbm\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr\" (UID: \"9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr"
Sep 05 19:52:47.657187 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.657160    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c09d415c-bc67-4c32-82f5-ec6490dd3d7b-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx\" (UID: \"c09d415c-bc67-4c32-82f5-ec6490dd3d7b\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx"
Sep 05 19:52:47.657187 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.657189    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr\" (UID: \"9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr"
Sep 05 19:52:47.657187 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.657211    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbc01914-1b88-49e5-a2f9-43625bd7a214-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr\" (UID: \"dbc01914-1b88-49e5-a2f9-43625bd7a214\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr"
Sep 05 19:52:47.657429 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.657269    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbc01914-1b88-49e5-a2f9-43625bd7a214-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr\" (UID: \"dbc01914-1b88-49e5-a2f9-43625bd7a214\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr"
Sep 05 19:52:47.657429 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.657296    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2srm\" (UniqueName: \"kubernetes.io/projected/dbc01914-1b88-49e5-a2f9-43625bd7a214-kube-api-access-b2srm\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr\" (UID: \"dbc01914-1b88-49e5-a2f9-43625bd7a214\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr"
Sep 05 19:52:47.657429 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.657331    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c09d415c-bc67-4c32-82f5-ec6490dd3d7b-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx\" (UID: \"c09d415c-bc67-4c32-82f5-ec6490dd3d7b\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx"
Sep 05 19:52:47.657748 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.657728    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbc01914-1b88-49e5-a2f9-43625bd7a214-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr\" (UID: \"dbc01914-1b88-49e5-a2f9-43625bd7a214\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr"
Sep 05 19:52:47.657861 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.657815    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr\" (UID: \"9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr"
Sep 05 19:52:47.657861 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.657810    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbc01914-1b88-49e5-a2f9-43625bd7a214-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr\" (UID: \"dbc01914-1b88-49e5-a2f9-43625bd7a214\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr"
Sep 05 19:52:47.665606 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.665577    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvhbm\" (UniqueName: \"kubernetes.io/projected/9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99-kube-api-access-vvhbm\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr\" (UID: \"9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr"
Sep 05 19:52:47.665972 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.665948    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2srm\" (UniqueName: \"kubernetes.io/projected/dbc01914-1b88-49e5-a2f9-43625bd7a214-kube-api-access-b2srm\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr\" (UID: \"dbc01914-1b88-49e5-a2f9-43625bd7a214\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr"
Sep 05 19:52:47.704464 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.704429    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr"
Sep 05 19:52:47.758728 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.758694    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cktls\" (UniqueName: \"kubernetes.io/projected/c09d415c-bc67-4c32-82f5-ec6490dd3d7b-kube-api-access-cktls\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx\" (UID: \"c09d415c-bc67-4c32-82f5-ec6490dd3d7b\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx"
Sep 05 19:52:47.758882 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.758772    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c09d415c-bc67-4c32-82f5-ec6490dd3d7b-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx\" (UID: \"c09d415c-bc67-4c32-82f5-ec6490dd3d7b\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx"
Sep 05 19:52:47.758882 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.758838    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c09d415c-bc67-4c32-82f5-ec6490dd3d7b-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx\" (UID: \"c09d415c-bc67-4c32-82f5-ec6490dd3d7b\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx"
Sep 05 19:52:47.759385 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.759309    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c09d415c-bc67-4c32-82f5-ec6490dd3d7b-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx\" (UID: \"c09d415c-bc67-4c32-82f5-ec6490dd3d7b\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx"
Sep 05 19:52:47.759469 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.759411    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c09d415c-bc67-4c32-82f5-ec6490dd3d7b-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx\" (UID: \"c09d415c-bc67-4c32-82f5-ec6490dd3d7b\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx"
Sep 05 19:52:47.771312 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.771290    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cktls\" (UniqueName: \"kubernetes.io/projected/c09d415c-bc67-4c32-82f5-ec6490dd3d7b-kube-api-access-cktls\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx\" (UID: \"c09d415c-bc67-4c32-82f5-ec6490dd3d7b\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx"
Sep 05 19:52:47.796453 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.796425    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr"
Sep 05 19:52:47.897648 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.897601    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx"
Sep 05 19:52:47.915456 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.915429    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc"]
Sep 05 19:52:47.917330 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:52:47.917307    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741f53bb_bad1_4881_a680_7fc4b1a20fd0.slice/crio-d41d1c1244eadaf9676758a30824a8045190c93fce6762cce3b9faaa8ec0cfb7 WatchSource:0}: Error finding container d41d1c1244eadaf9676758a30824a8045190c93fce6762cce3b9faaa8ec0cfb7: Status 404 returned error can't find the container with id d41d1c1244eadaf9676758a30824a8045190c93fce6762cce3b9faaa8ec0cfb7
Sep 05 19:52:47.982973 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:47.982944    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr"]
Sep 05 19:52:47.984274 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:52:47.984251    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb4bcf0_dafc_41a1_b8d5_b75cfd42be99.slice/crio-28304b8650536285f7407f89067e406c6f21bb0381400e3cd8e929dfa372f963 WatchSource:0}: Error finding container 28304b8650536285f7407f89067e406c6f21bb0381400e3cd8e929dfa372f963: Status 404 returned error can't find the container with id 28304b8650536285f7407f89067e406c6f21bb0381400e3cd8e929dfa372f963
Sep 05 19:52:48.066950 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:48.066927    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr"]
Sep 05 19:52:48.140143 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:52:48.140052    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc01914_1b88_49e5_a2f9_43625bd7a214.slice/crio-60aaced7e59f5cdaf60c797bcfcd7cac32a8df7e70c76e6c856c4a3645d7b775 WatchSource:0}: Error finding container 60aaced7e59f5cdaf60c797bcfcd7cac32a8df7e70c76e6c856c4a3645d7b775: Status 404 returned error can't find the container with id 60aaced7e59f5cdaf60c797bcfcd7cac32a8df7e70c76e6c856c4a3645d7b775
Sep 05 19:52:48.172632 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:48.172608    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx"]
Sep 05 19:52:48.173957 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:52:48.173934    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc09d415c_bc67_4c32_82f5_ec6490dd3d7b.slice/crio-e27936d48405ae565e61a7d33fda106fc43090a8de7a0ad692bd754eb420d410 WatchSource:0}: Error finding container e27936d48405ae565e61a7d33fda106fc43090a8de7a0ad692bd754eb420d410: Status 404 returned error can't find the container with id e27936d48405ae565e61a7d33fda106fc43090a8de7a0ad692bd754eb420d410
Sep 05 19:52:48.753739 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:48.753707    2583 generic.go:356] "Generic (PLEG): container finished" podID="741f53bb-bad1-4881-a680-7fc4b1a20fd0" containerID="532c30153590d5239e6877c15a04f03bed53aac9959251e4ec8dc16bcfb68880" exitCode=0
Sep 05 19:52:48.754122 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:48.753790    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc" event={"ID":"741f53bb-bad1-4881-a680-7fc4b1a20fd0","Type":"ContainerDied","Data":"532c30153590d5239e6877c15a04f03bed53aac9959251e4ec8dc16bcfb68880"}
Sep 05 19:52:48.754122 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:48.753876    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc" event={"ID":"741f53bb-bad1-4881-a680-7fc4b1a20fd0","Type":"ContainerStarted","Data":"d41d1c1244eadaf9676758a30824a8045190c93fce6762cce3b9faaa8ec0cfb7"}
Sep 05 19:52:48.755527 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:48.755495    2583 generic.go:356] "Generic (PLEG): container finished" podID="9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99" containerID="a3f37e75ff6add5e76098719dc26ce498ce54533dc92e23ff51dda78f7012029" exitCode=0
Sep 05 19:52:48.755610 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:48.755536    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr" event={"ID":"9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99","Type":"ContainerDied","Data":"a3f37e75ff6add5e76098719dc26ce498ce54533dc92e23ff51dda78f7012029"}
Sep 05 19:52:48.755610 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:48.755582    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr" event={"ID":"9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99","Type":"ContainerStarted","Data":"28304b8650536285f7407f89067e406c6f21bb0381400e3cd8e929dfa372f963"}
Sep 05 19:52:48.757169 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:48.757151    2583 generic.go:356] "Generic (PLEG): container finished" podID="dbc01914-1b88-49e5-a2f9-43625bd7a214" containerID="513c8602300b322aa0fd4713833ec5a6dec0032d78bd41d4409d741cfdbde52e" exitCode=0
Sep 05 19:52:48.757255 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:48.757230    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr" event={"ID":"dbc01914-1b88-49e5-a2f9-43625bd7a214","Type":"ContainerDied","Data":"513c8602300b322aa0fd4713833ec5a6dec0032d78bd41d4409d741cfdbde52e"}
Sep 05 19:52:48.757316 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:48.757268    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr" event={"ID":"dbc01914-1b88-49e5-a2f9-43625bd7a214","Type":"ContainerStarted","Data":"60aaced7e59f5cdaf60c797bcfcd7cac32a8df7e70c76e6c856c4a3645d7b775"}
Sep 05 19:52:48.758798 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:48.758774    2583 generic.go:356] "Generic (PLEG): container finished" podID="c09d415c-bc67-4c32-82f5-ec6490dd3d7b" containerID="e2f55caab13604f5e31c97445e9db68da23c29eafd33949bef2e06cb8a56eeee" exitCode=0
Sep 05 19:52:48.758882 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:48.758822    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx" event={"ID":"c09d415c-bc67-4c32-82f5-ec6490dd3d7b","Type":"ContainerDied","Data":"e2f55caab13604f5e31c97445e9db68da23c29eafd33949bef2e06cb8a56eeee"}
Sep 05 19:52:48.758882 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:48.758848    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx" event={"ID":"c09d415c-bc67-4c32-82f5-ec6490dd3d7b","Type":"ContainerStarted","Data":"e27936d48405ae565e61a7d33fda106fc43090a8de7a0ad692bd754eb420d410"}
Sep 05 19:52:50.771081 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:50.770991    2583 generic.go:356] "Generic (PLEG): container finished" podID="741f53bb-bad1-4881-a680-7fc4b1a20fd0" containerID="e5466b290d195eddd1dd859f8ebe918d06178c4b439c870239b332ca247ccb12" exitCode=0
Sep 05 19:52:50.771472 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:50.771077    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc" event={"ID":"741f53bb-bad1-4881-a680-7fc4b1a20fd0","Type":"ContainerDied","Data":"e5466b290d195eddd1dd859f8ebe918d06178c4b439c870239b332ca247ccb12"}
Sep 05 19:52:50.773065 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:50.773036    2583 generic.go:356] "Generic (PLEG): container finished" podID="9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99" containerID="d9ec74ddd18139fd4948a7ad715ff08d46632542dbc3b0c7247e91f53cf9d7b2" exitCode=0
Sep 05 19:52:50.773210 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:50.773083    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr" event={"ID":"9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99","Type":"ContainerDied","Data":"d9ec74ddd18139fd4948a7ad715ff08d46632542dbc3b0c7247e91f53cf9d7b2"}
Sep 05 19:52:50.775060 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:50.775035    2583 generic.go:356] "Generic (PLEG): container finished" podID="dbc01914-1b88-49e5-a2f9-43625bd7a214" containerID="b684888db3cda6893224a317cb1729077b65ce9048db9d7635ed75c79aa32d40" exitCode=0
Sep 05 19:52:50.775162 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:50.775133    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr" event={"ID":"dbc01914-1b88-49e5-a2f9-43625bd7a214","Type":"ContainerDied","Data":"b684888db3cda6893224a317cb1729077b65ce9048db9d7635ed75c79aa32d40"}
Sep 05 19:52:50.777183 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:50.777160    2583 generic.go:356] "Generic (PLEG): container finished" podID="c09d415c-bc67-4c32-82f5-ec6490dd3d7b" containerID="446b2e8bb75854b1ed88f60d57d763f7a19ea7b620e6f797ec88b3e2c2f65ead" exitCode=0
Sep 05 19:52:50.777292 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:50.777206    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx" event={"ID":"c09d415c-bc67-4c32-82f5-ec6490dd3d7b","Type":"ContainerDied","Data":"446b2e8bb75854b1ed88f60d57d763f7a19ea7b620e6f797ec88b3e2c2f65ead"}
Sep 05 19:52:51.784295 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:51.784252    2583 generic.go:356] "Generic (PLEG): container finished" podID="dbc01914-1b88-49e5-a2f9-43625bd7a214" containerID="9778279787ed7e42e3f4a0a950a8633e979b9cd59e748f47ccd11d1fff047b96" exitCode=0
Sep 05 19:52:51.784686 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:51.784334    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr" event={"ID":"dbc01914-1b88-49e5-a2f9-43625bd7a214","Type":"ContainerDied","Data":"9778279787ed7e42e3f4a0a950a8633e979b9cd59e748f47ccd11d1fff047b96"}
Sep 05 19:52:51.786458 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:51.786436    2583 generic.go:356] "Generic (PLEG): container finished" podID="c09d415c-bc67-4c32-82f5-ec6490dd3d7b" containerID="68edc625673a65681ac557ed00a4ed69beb54a6c71eb19729cff42a0e449974d" exitCode=0
Sep 05 19:52:51.786552 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:51.786517    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx" event={"ID":"c09d415c-bc67-4c32-82f5-ec6490dd3d7b","Type":"ContainerDied","Data":"68edc625673a65681ac557ed00a4ed69beb54a6c71eb19729cff42a0e449974d"}
Sep 05 19:52:51.788571 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:51.788550    2583 generic.go:356] "Generic (PLEG): container finished" podID="741f53bb-bad1-4881-a680-7fc4b1a20fd0" containerID="d8338f2db64163c026e1b9972cb4d1427adce752ef49af49f5c0cf83ba44316a" exitCode=0
Sep 05 19:52:51.788682 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:51.788621    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc" event={"ID":"741f53bb-bad1-4881-a680-7fc4b1a20fd0","Type":"ContainerDied","Data":"d8338f2db64163c026e1b9972cb4d1427adce752ef49af49f5c0cf83ba44316a"}
Sep 05 19:52:51.790640 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:51.790621    2583 generic.go:356] "Generic (PLEG): container finished" podID="9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99" containerID="ce46ea19ad05756123af8ab9956c3314f12ffd8a2d566d16c805b318d11ebc91" exitCode=0
Sep 05 19:52:51.790715 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:51.790661    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr" event={"ID":"9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99","Type":"ContainerDied","Data":"ce46ea19ad05756123af8ab9956c3314f12ffd8a2d566d16c805b318d11ebc91"}
Sep 05 19:52:52.954322 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:52.954295    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr"
Sep 05 19:52:52.976752 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:52.976725    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx"
Sep 05 19:52:52.980125 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:52.980095    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc"
Sep 05 19:52:52.984747 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:52.984727    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr"
Sep 05 19:52:53.105409 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.105378    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2srm\" (UniqueName: \"kubernetes.io/projected/dbc01914-1b88-49e5-a2f9-43625bd7a214-kube-api-access-b2srm\") pod \"dbc01914-1b88-49e5-a2f9-43625bd7a214\" (UID: \"dbc01914-1b88-49e5-a2f9-43625bd7a214\") "
Sep 05 19:52:53.105583 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.105424    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/741f53bb-bad1-4881-a680-7fc4b1a20fd0-util\") pod \"741f53bb-bad1-4881-a680-7fc4b1a20fd0\" (UID: \"741f53bb-bad1-4881-a680-7fc4b1a20fd0\") "
Sep 05 19:52:53.105583 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.105445    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cktls\" (UniqueName: \"kubernetes.io/projected/c09d415c-bc67-4c32-82f5-ec6490dd3d7b-kube-api-access-cktls\") pod \"c09d415c-bc67-4c32-82f5-ec6490dd3d7b\" (UID: \"c09d415c-bc67-4c32-82f5-ec6490dd3d7b\") "
Sep 05 19:52:53.105583 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.105477    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/741f53bb-bad1-4881-a680-7fc4b1a20fd0-bundle\") pod \"741f53bb-bad1-4881-a680-7fc4b1a20fd0\" (UID: \"741f53bb-bad1-4881-a680-7fc4b1a20fd0\") "
Sep 05 19:52:53.105583 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.105496    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nwl5\" (UniqueName: \"kubernetes.io/projected/741f53bb-bad1-4881-a680-7fc4b1a20fd0-kube-api-access-6nwl5\") pod \"741f53bb-bad1-4881-a680-7fc4b1a20fd0\" (UID: \"741f53bb-bad1-4881-a680-7fc4b1a20fd0\") "
Sep 05 19:52:53.105583 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.105511    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvhbm\" (UniqueName: \"kubernetes.io/projected/9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99-kube-api-access-vvhbm\") pod \"9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99\" (UID: \"9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99\") "
Sep 05 19:52:53.105583 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.105532    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99-util\") pod \"9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99\" (UID: \"9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99\") "
Sep 05 19:52:53.105946 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.105770    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbc01914-1b88-49e5-a2f9-43625bd7a214-bundle\") pod \"dbc01914-1b88-49e5-a2f9-43625bd7a214\" (UID: \"dbc01914-1b88-49e5-a2f9-43625bd7a214\") "
Sep 05 19:52:53.105946 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.105813    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c09d415c-bc67-4c32-82f5-ec6490dd3d7b-bundle\") pod \"c09d415c-bc67-4c32-82f5-ec6490dd3d7b\" (UID: \"c09d415c-bc67-4c32-82f5-ec6490dd3d7b\") "
Sep 05 19:52:53.105946 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.105854    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c09d415c-bc67-4c32-82f5-ec6490dd3d7b-util\") pod \"c09d415c-bc67-4c32-82f5-ec6490dd3d7b\" (UID: \"c09d415c-bc67-4c32-82f5-ec6490dd3d7b\") "
Sep 05 19:52:53.105946 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.105911    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbc01914-1b88-49e5-a2f9-43625bd7a214-util\") pod \"dbc01914-1b88-49e5-a2f9-43625bd7a214\" (UID: \"dbc01914-1b88-49e5-a2f9-43625bd7a214\") "
Sep 05 19:52:53.105946 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.105937    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99-bundle\") pod \"9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99\" (UID: \"9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99\") "
Sep 05 19:52:53.106708 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.106679    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99-bundle" (OuterVolumeSpecName: "bundle") pod "9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99" (UID: "9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:52:53.106917 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.106894    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09d415c-bc67-4c32-82f5-ec6490dd3d7b-bundle" (OuterVolumeSpecName: "bundle") pod "c09d415c-bc67-4c32-82f5-ec6490dd3d7b" (UID: "c09d415c-bc67-4c32-82f5-ec6490dd3d7b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:52:53.108586 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.107378    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc01914-1b88-49e5-a2f9-43625bd7a214-bundle" (OuterVolumeSpecName: "bundle") pod "dbc01914-1b88-49e5-a2f9-43625bd7a214" (UID: "dbc01914-1b88-49e5-a2f9-43625bd7a214"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:52:53.108586 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.107881    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741f53bb-bad1-4881-a680-7fc4b1a20fd0-kube-api-access-6nwl5" (OuterVolumeSpecName: "kube-api-access-6nwl5") pod "741f53bb-bad1-4881-a680-7fc4b1a20fd0" (UID: "741f53bb-bad1-4881-a680-7fc4b1a20fd0"). InnerVolumeSpecName "kube-api-access-6nwl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:52:53.108586 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.108020    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc01914-1b88-49e5-a2f9-43625bd7a214-kube-api-access-b2srm" (OuterVolumeSpecName: "kube-api-access-b2srm") pod "dbc01914-1b88-49e5-a2f9-43625bd7a214" (UID: "dbc01914-1b88-49e5-a2f9-43625bd7a214"). InnerVolumeSpecName "kube-api-access-b2srm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:52:53.108586 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.108526    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741f53bb-bad1-4881-a680-7fc4b1a20fd0-bundle" (OuterVolumeSpecName: "bundle") pod "741f53bb-bad1-4881-a680-7fc4b1a20fd0" (UID: "741f53bb-bad1-4881-a680-7fc4b1a20fd0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:52:53.109002 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.108984    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99-kube-api-access-vvhbm" (OuterVolumeSpecName: "kube-api-access-vvhbm") pod "9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99" (UID: "9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99"). InnerVolumeSpecName "kube-api-access-vvhbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:52:53.109386 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.109371    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c09d415c-bc67-4c32-82f5-ec6490dd3d7b-kube-api-access-cktls" (OuterVolumeSpecName: "kube-api-access-cktls") pod "c09d415c-bc67-4c32-82f5-ec6490dd3d7b" (UID: "c09d415c-bc67-4c32-82f5-ec6490dd3d7b"). InnerVolumeSpecName "kube-api-access-cktls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:52:53.112974 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.112951    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99-util" (OuterVolumeSpecName: "util") pod "9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99" (UID: "9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:52:53.113131 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.113095    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741f53bb-bad1-4881-a680-7fc4b1a20fd0-util" (OuterVolumeSpecName: "util") pod "741f53bb-bad1-4881-a680-7fc4b1a20fd0" (UID: "741f53bb-bad1-4881-a680-7fc4b1a20fd0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:52:53.115896 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.115874    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09d415c-bc67-4c32-82f5-ec6490dd3d7b-util" (OuterVolumeSpecName: "util") pod "c09d415c-bc67-4c32-82f5-ec6490dd3d7b" (UID: "c09d415c-bc67-4c32-82f5-ec6490dd3d7b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:52:53.116300 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.116286    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc01914-1b88-49e5-a2f9-43625bd7a214-util" (OuterVolumeSpecName: "util") pod "dbc01914-1b88-49e5-a2f9-43625bd7a214" (UID: "dbc01914-1b88-49e5-a2f9-43625bd7a214"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:52:53.206826 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.206789    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/741f53bb-bad1-4881-a680-7fc4b1a20fd0-bundle\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:52:53.206826 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.206822    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nwl5\" (UniqueName: \"kubernetes.io/projected/741f53bb-bad1-4881-a680-7fc4b1a20fd0-kube-api-access-6nwl5\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:52:53.207013 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.206837    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvhbm\" (UniqueName: \"kubernetes.io/projected/9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99-kube-api-access-vvhbm\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:52:53.207013 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.206848    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99-util\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:52:53.207013 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.206859    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbc01914-1b88-49e5-a2f9-43625bd7a214-bundle\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:52:53.207013 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.206870    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c09d415c-bc67-4c32-82f5-ec6490dd3d7b-bundle\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:52:53.207013 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.206880    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c09d415c-bc67-4c32-82f5-ec6490dd3d7b-util\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:52:53.207013 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.206889    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbc01914-1b88-49e5-a2f9-43625bd7a214-util\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:52:53.207013 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.206899    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99-bundle\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:52:53.207013 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.206910    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2srm\" (UniqueName: \"kubernetes.io/projected/dbc01914-1b88-49e5-a2f9-43625bd7a214-kube-api-access-b2srm\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:52:53.207013 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.206921    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/741f53bb-bad1-4881-a680-7fc4b1a20fd0-util\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:52:53.207013 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.206932    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cktls\" (UniqueName: \"kubernetes.io/projected/c09d415c-bc67-4c32-82f5-ec6490dd3d7b-kube-api-access-cktls\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:52:53.802656 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.802566    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr" event={"ID":"9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99","Type":"ContainerDied","Data":"28304b8650536285f7407f89067e406c6f21bb0381400e3cd8e929dfa372f963"}
Sep 05 19:52:53.802656 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.802623    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28304b8650536285f7407f89067e406c6f21bb0381400e3cd8e929dfa372f963"
Sep 05 19:52:53.802656 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.802576    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cb98qr"
Sep 05 19:52:53.804656 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.804631    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr"
Sep 05 19:52:53.804838 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.804667    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbgq6cr" event={"ID":"dbc01914-1b88-49e5-a2f9-43625bd7a214","Type":"ContainerDied","Data":"60aaced7e59f5cdaf60c797bcfcd7cac32a8df7e70c76e6c856c4a3645d7b775"}
Sep 05 19:52:53.804838 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.804703    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60aaced7e59f5cdaf60c797bcfcd7cac32a8df7e70c76e6c856c4a3645d7b775"
Sep 05 19:52:53.806999 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.806978    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx"
Sep 05 19:52:53.806999 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.806987    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0mhgpx" event={"ID":"c09d415c-bc67-4c32-82f5-ec6490dd3d7b","Type":"ContainerDied","Data":"e27936d48405ae565e61a7d33fda106fc43090a8de7a0ad692bd754eb420d410"}
Sep 05 19:52:53.807157 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.807014    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e27936d48405ae565e61a7d33fda106fc43090a8de7a0ad692bd754eb420d410"
Sep 05 19:52:53.809045 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.809023    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc" event={"ID":"741f53bb-bad1-4881-a680-7fc4b1a20fd0","Type":"ContainerDied","Data":"d41d1c1244eadaf9676758a30824a8045190c93fce6762cce3b9faaa8ec0cfb7"}
Sep 05 19:52:53.809142 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.809055    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d41d1c1244eadaf9676758a30824a8045190c93fce6762cce3b9faaa8ec0cfb7"
Sep 05 19:52:53.809142 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:53.809027    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053xmckc"
Sep 05 19:52:59.462332 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462298    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7n7z"]
Sep 05 19:52:59.462685 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462611    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c09d415c-bc67-4c32-82f5-ec6490dd3d7b" containerName="extract"
Sep 05 19:52:59.462685 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462622    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c09d415c-bc67-4c32-82f5-ec6490dd3d7b" containerName="extract"
Sep 05 19:52:59.462685 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462630    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbc01914-1b88-49e5-a2f9-43625bd7a214" containerName="util"
Sep 05 19:52:59.462685 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462636    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbc01914-1b88-49e5-a2f9-43625bd7a214" containerName="util"
Sep 05 19:52:59.462685 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462642    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbc01914-1b88-49e5-a2f9-43625bd7a214" containerName="pull"
Sep 05 19:52:59.462685 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462647    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbc01914-1b88-49e5-a2f9-43625bd7a214" containerName="pull"
Sep 05 19:52:59.462685 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462657    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99" containerName="extract"
Sep 05 19:52:59.462685 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462666    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99" containerName="extract"
Sep 05 19:52:59.462685 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462675    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99" containerName="pull"
Sep 05 19:52:59.462685 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462680    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99" containerName="pull"
Sep 05 19:52:59.462685 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462687    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="741f53bb-bad1-4881-a680-7fc4b1a20fd0" containerName="pull"
Sep 05 19:52:59.462999 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462693    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="741f53bb-bad1-4881-a680-7fc4b1a20fd0" containerName="pull"
Sep 05 19:52:59.462999 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462698    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="741f53bb-bad1-4881-a680-7fc4b1a20fd0" containerName="util"
Sep 05 19:52:59.462999 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462703    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="741f53bb-bad1-4881-a680-7fc4b1a20fd0" containerName="util"
Sep 05 19:52:59.462999 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462713    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99" containerName="util"
Sep 05 19:52:59.462999 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462718    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99" containerName="util"
Sep 05 19:52:59.462999 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462727    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="741f53bb-bad1-4881-a680-7fc4b1a20fd0" containerName="extract"
Sep 05 19:52:59.462999 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462733    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="741f53bb-bad1-4881-a680-7fc4b1a20fd0" containerName="extract"
Sep 05 19:52:59.462999 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462748    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c09d415c-bc67-4c32-82f5-ec6490dd3d7b" containerName="util"
Sep 05 19:52:59.462999 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462753    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c09d415c-bc67-4c32-82f5-ec6490dd3d7b" containerName="util"
Sep 05 19:52:59.462999 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462760    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c09d415c-bc67-4c32-82f5-ec6490dd3d7b" containerName="pull"
Sep 05 19:52:59.462999 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462764    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c09d415c-bc67-4c32-82f5-ec6490dd3d7b" containerName="pull"
Sep 05 19:52:59.462999 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462773    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbc01914-1b88-49e5-a2f9-43625bd7a214" containerName="extract"
Sep 05 19:52:59.462999 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462778    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbc01914-1b88-49e5-a2f9-43625bd7a214" containerName="extract"
Sep 05 19:52:59.462999 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462829    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbc01914-1b88-49e5-a2f9-43625bd7a214" containerName="extract"
Sep 05 19:52:59.462999 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462837    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c09d415c-bc67-4c32-82f5-ec6490dd3d7b" containerName="extract"
Sep 05 19:52:59.462999 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462843    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bb4bcf0-dafc-41a1-b8d5-b75cfd42be99" containerName="extract"
Sep 05 19:52:59.462999 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.462851    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="741f53bb-bad1-4881-a680-7fc4b1a20fd0" containerName="extract"
Sep 05 19:52:59.466919 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.466904    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7n7z"
Sep 05 19:52:59.469219 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.469197    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 05 19:52:59.469367 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.469350    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 05 19:52:59.469622 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.469609    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-jxjpb\""
Sep 05 19:52:59.475729 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.475705    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7n7z"]
Sep 05 19:52:59.564746 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.564716    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5xc5\" (UniqueName: \"kubernetes.io/projected/a067b15a-1fd3-4969-93ff-49a56a5e376c-kube-api-access-h5xc5\") pod \"limitador-operator-controller-manager-6c549b6d85-v7n7z\" (UID: \"a067b15a-1fd3-4969-93ff-49a56a5e376c\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7n7z"
Sep 05 19:52:59.665772 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.665730    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5xc5\" (UniqueName: \"kubernetes.io/projected/a067b15a-1fd3-4969-93ff-49a56a5e376c-kube-api-access-h5xc5\") pod \"limitador-operator-controller-manager-6c549b6d85-v7n7z\" (UID: \"a067b15a-1fd3-4969-93ff-49a56a5e376c\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7n7z"
Sep 05 19:52:59.674657 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.674624    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5xc5\" (UniqueName: \"kubernetes.io/projected/a067b15a-1fd3-4969-93ff-49a56a5e376c-kube-api-access-h5xc5\") pod \"limitador-operator-controller-manager-6c549b6d85-v7n7z\" (UID: \"a067b15a-1fd3-4969-93ff-49a56a5e376c\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7n7z"
Sep 05 19:52:59.777854 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:52:59.777779    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7n7z"
Sep 05 19:53:00.050375 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:00.050350    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7n7z"]
Sep 05 19:53:00.051974 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:53:00.051947    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda067b15a_1fd3_4969_93ff_49a56a5e376c.slice/crio-28f676ced6804407bf82ee70ba404ca6ff71f99663b7b1e5ce4d2b641b706cb8 WatchSource:0}: Error finding container 28f676ced6804407bf82ee70ba404ca6ff71f99663b7b1e5ce4d2b641b706cb8: Status 404 returned error can't find the container with id 28f676ced6804407bf82ee70ba404ca6ff71f99663b7b1e5ce4d2b641b706cb8
Sep 05 19:53:00.848159 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:00.848104    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7n7z" event={"ID":"a067b15a-1fd3-4969-93ff-49a56a5e376c","Type":"ContainerStarted","Data":"28f676ced6804407bf82ee70ba404ca6ff71f99663b7b1e5ce4d2b641b706cb8"}
Sep 05 19:53:01.898284 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:01.898247    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-hjl4w"]
Sep 05 19:53:01.906963 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:01.906938    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-hjl4w"
Sep 05 19:53:01.909536 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:01.909510    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-9hgqr\""
Sep 05 19:53:01.915209 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:01.915185    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-hjl4w"]
Sep 05 19:53:01.982859 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:01.982818    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqpjn\" (UniqueName: \"kubernetes.io/projected/351f07d9-27e1-436e-81fc-f10f78cf16d0-kube-api-access-qqpjn\") pod \"authorino-operator-68d88948d8-hjl4w\" (UID: \"351f07d9-27e1-436e-81fc-f10f78cf16d0\") " pod="kuadrant-system/authorino-operator-68d88948d8-hjl4w"
Sep 05 19:53:02.083431 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:02.083392    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqpjn\" (UniqueName: \"kubernetes.io/projected/351f07d9-27e1-436e-81fc-f10f78cf16d0-kube-api-access-qqpjn\") pod \"authorino-operator-68d88948d8-hjl4w\" (UID: \"351f07d9-27e1-436e-81fc-f10f78cf16d0\") " pod="kuadrant-system/authorino-operator-68d88948d8-hjl4w"
Sep 05 19:53:02.094382 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:02.094354    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqpjn\" (UniqueName: \"kubernetes.io/projected/351f07d9-27e1-436e-81fc-f10f78cf16d0-kube-api-access-qqpjn\") pod \"authorino-operator-68d88948d8-hjl4w\" (UID: \"351f07d9-27e1-436e-81fc-f10f78cf16d0\") " pod="kuadrant-system/authorino-operator-68d88948d8-hjl4w"
Sep 05 19:53:02.221085 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:02.220991    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-hjl4w"
Sep 05 19:53:02.855312 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:02.855210    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-hjl4w"]
Sep 05 19:53:02.858262 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:53:02.858229    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351f07d9_27e1_436e_81fc_f10f78cf16d0.slice/crio-9220a412c2512906fbd1beb0e521e11e148984a6955a65408eef09dbd63b80e5 WatchSource:0}: Error finding container 9220a412c2512906fbd1beb0e521e11e148984a6955a65408eef09dbd63b80e5: Status 404 returned error can't find the container with id 9220a412c2512906fbd1beb0e521e11e148984a6955a65408eef09dbd63b80e5
Sep 05 19:53:02.860742 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:02.860714    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7n7z" event={"ID":"a067b15a-1fd3-4969-93ff-49a56a5e376c","Type":"ContainerStarted","Data":"1b3d20eaee778a0a249781698514a2a87779b6b511046ad8b06175140cb4fbd6"}
Sep 05 19:53:02.861027 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:02.860788    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7n7z"
Sep 05 19:53:02.889465 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:02.889406    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7n7z" podStartSLOduration=1.290187831 podStartE2EDuration="3.889388536s" totalImagesPullingTime="2.599200705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:52:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:53:02.886394954 +0000 UTC m=+581.916026858" watchObservedRunningTime="2026-09-05 19:53:02.889388536 +0000 UTC m=+581.919020443"
Sep 05 19:53:03.867496 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:03.867460    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-hjl4w" event={"ID":"351f07d9-27e1-436e-81fc-f10f78cf16d0","Type":"ContainerStarted","Data":"9220a412c2512906fbd1beb0e521e11e148984a6955a65408eef09dbd63b80e5"}
Sep 05 19:53:04.240388 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:04.240353    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-vpk9d"]
Sep 05 19:53:04.243897 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:04.243881    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vpk9d"
Sep 05 19:53:04.246301 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:04.246281    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 05 19:53:04.246523 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:04.246507    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-679l8\""
Sep 05 19:53:04.256882 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:04.256854    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-vpk9d"]
Sep 05 19:53:04.300303 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:04.300263    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hrqn\" (UniqueName: \"kubernetes.io/projected/f50c5f88-50ad-453a-b40e-593cff2208c4-kube-api-access-5hrqn\") pod \"dns-operator-controller-manager-5985564776-vpk9d\" (UID: \"f50c5f88-50ad-453a-b40e-593cff2208c4\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-vpk9d"
Sep 05 19:53:04.400709 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:04.400667    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hrqn\" (UniqueName: \"kubernetes.io/projected/f50c5f88-50ad-453a-b40e-593cff2208c4-kube-api-access-5hrqn\") pod \"dns-operator-controller-manager-5985564776-vpk9d\" (UID: \"f50c5f88-50ad-453a-b40e-593cff2208c4\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-vpk9d"
Sep 05 19:53:04.412124 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:04.412090    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hrqn\" (UniqueName: \"kubernetes.io/projected/f50c5f88-50ad-453a-b40e-593cff2208c4-kube-api-access-5hrqn\") pod \"dns-operator-controller-manager-5985564776-vpk9d\" (UID: \"f50c5f88-50ad-453a-b40e-593cff2208c4\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-vpk9d"
Sep 05 19:53:04.556228 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:04.556142    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vpk9d"
Sep 05 19:53:04.834591 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:04.834565    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-vpk9d"]
Sep 05 19:53:04.836276 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:53:04.836250    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf50c5f88_50ad_453a_b40e_593cff2208c4.slice/crio-a7dad975bf6d0f9e5bd6459869b261b8adea182f27208008f2340ca7e74188d0 WatchSource:0}: Error finding container a7dad975bf6d0f9e5bd6459869b261b8adea182f27208008f2340ca7e74188d0: Status 404 returned error can't find the container with id a7dad975bf6d0f9e5bd6459869b261b8adea182f27208008f2340ca7e74188d0
Sep 05 19:53:04.875009 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:04.874972    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-hjl4w" event={"ID":"351f07d9-27e1-436e-81fc-f10f78cf16d0","Type":"ContainerStarted","Data":"1983fc0f65795aa746b64c25620eb6f4c5ac62d0935e264814d2e5822022ead7"}
Sep 05 19:53:04.875459 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:04.875044    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-hjl4w"
Sep 05 19:53:04.876453 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:04.876428    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vpk9d" event={"ID":"f50c5f88-50ad-453a-b40e-593cff2208c4","Type":"ContainerStarted","Data":"a7dad975bf6d0f9e5bd6459869b261b8adea182f27208008f2340ca7e74188d0"}
Sep 05 19:53:04.895356 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:04.895310    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-hjl4w" podStartSLOduration=2.587859289 podStartE2EDuration="3.89529295s" totalImagesPullingTime="1.307433661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:53:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:53:04.893688202 +0000 UTC m=+583.923320108" watchObservedRunningTime="2026-09-05 19:53:04.89529295 +0000 UTC m=+583.924924857"
Sep 05 19:53:07.893619 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:07.893581    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vpk9d" event={"ID":"f50c5f88-50ad-453a-b40e-593cff2208c4","Type":"ContainerStarted","Data":"54e384f9ef7e388bd5279bf98141b319c88c49848d79fa317dc6ae9daf7af8ec"}
Sep 05 19:53:07.894023 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:07.893697    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vpk9d"
Sep 05 19:53:07.913024 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:07.912978    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vpk9d" podStartSLOduration=1.483189162 podStartE2EDuration="3.912964801s" totalImagesPullingTime="2.429775639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:53:07.911440175 +0000 UTC m=+586.941072079" watchObservedRunningTime="2026-09-05 19:53:07.912964801 +0000 UTC m=+586.942596707"
Sep 05 19:53:08.573751 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:08.573717    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9"]
Sep 05 19:53:08.576695 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:08.576679    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9"
Sep 05 19:53:08.580314 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:08.580290    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tq2z7\""
Sep 05 19:53:08.591689 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:08.591669    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9"]
Sep 05 19:53:08.634177 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:08.634150    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqhfj\" (UniqueName: \"kubernetes.io/projected/a01ebc67-0f15-479d-b91f-02d146697eea-kube-api-access-bqhfj\") pod \"kuadrant-operator-controller-manager-58fc56757c-xqxt9\" (UID: \"a01ebc67-0f15-479d-b91f-02d146697eea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9"
Sep 05 19:53:08.634335 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:08.634209    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a01ebc67-0f15-479d-b91f-02d146697eea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-xqxt9\" (UID: \"a01ebc67-0f15-479d-b91f-02d146697eea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9"
Sep 05 19:53:08.734816 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:08.734785    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a01ebc67-0f15-479d-b91f-02d146697eea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-xqxt9\" (UID: \"a01ebc67-0f15-479d-b91f-02d146697eea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9"
Sep 05 19:53:08.734973 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:08.734854    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqhfj\" (UniqueName: \"kubernetes.io/projected/a01ebc67-0f15-479d-b91f-02d146697eea-kube-api-access-bqhfj\") pod \"kuadrant-operator-controller-manager-58fc56757c-xqxt9\" (UID: \"a01ebc67-0f15-479d-b91f-02d146697eea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9"
Sep 05 19:53:08.735150 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:08.735132    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a01ebc67-0f15-479d-b91f-02d146697eea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-xqxt9\" (UID: \"a01ebc67-0f15-479d-b91f-02d146697eea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9"
Sep 05 19:53:08.742758 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:08.742735    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqhfj\" (UniqueName: \"kubernetes.io/projected/a01ebc67-0f15-479d-b91f-02d146697eea-kube-api-access-bqhfj\") pod \"kuadrant-operator-controller-manager-58fc56757c-xqxt9\" (UID: \"a01ebc67-0f15-479d-b91f-02d146697eea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9"
Sep 05 19:53:08.886878 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:08.886791    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9"
Sep 05 19:53:09.165251 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:09.165220    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9"]
Sep 05 19:53:09.167238 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:53:09.167209    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda01ebc67_0f15_479d_b91f_02d146697eea.slice/crio-1a700a2dd26e0372a8b9f5c7b5df6541570fb4966d32cd3cc1dc45621e218b5c WatchSource:0}: Error finding container 1a700a2dd26e0372a8b9f5c7b5df6541570fb4966d32cd3cc1dc45621e218b5c: Status 404 returned error can't find the container with id 1a700a2dd26e0372a8b9f5c7b5df6541570fb4966d32cd3cc1dc45621e218b5c
Sep 05 19:53:09.904791 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:09.904757    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9" event={"ID":"a01ebc67-0f15-479d-b91f-02d146697eea","Type":"ContainerStarted","Data":"1a700a2dd26e0372a8b9f5c7b5df6541570fb4966d32cd3cc1dc45621e218b5c"}
Sep 05 19:53:13.870858 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:13.870822    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7n7z"
Sep 05 19:53:15.883821 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:15.883787    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-hjl4w"
Sep 05 19:53:15.937715 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:15.937684    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9" event={"ID":"a01ebc67-0f15-479d-b91f-02d146697eea","Type":"ContainerStarted","Data":"759a782e8d8c0138d9743affbc5e7469101ad3da25722c56369e30e7d152effa"}
Sep 05 19:53:15.937874 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:15.937792    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9"
Sep 05 19:53:15.958046 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:15.957993    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9" podStartSLOduration=2.069924076 podStartE2EDuration="7.957977842s" totalImagesPullingTime="5.888053766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:53:15.955152634 +0000 UTC m=+594.984784540" watchObservedRunningTime="2026-09-05 19:53:15.957977842 +0000 UTC m=+594.987609746"
Sep 05 19:53:17.625658 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:17.625618    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-98s8s"]
Sep 05 19:53:17.629155 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:17.629134    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-98s8s"
Sep 05 19:53:17.631220 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:17.631199    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 05 19:53:17.631338 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:17.631301    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 05 19:53:17.631429 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:17.631411    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-m9t9m\""
Sep 05 19:53:17.635648 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:17.635629    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-98s8s"]
Sep 05 19:53:17.709610 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:17.709573    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/aa515af9-3fdb-403d-bd09-837323c156f6-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-98s8s\" (UID: \"aa515af9-3fdb-403d-bd09-837323c156f6\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-98s8s"
Sep 05 19:53:17.709770 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:17.709631    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa515af9-3fdb-403d-bd09-837323c156f6-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-98s8s\" (UID: \"aa515af9-3fdb-403d-bd09-837323c156f6\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-98s8s"
Sep 05 19:53:17.709770 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:17.709650    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ffgv\" (UniqueName: \"kubernetes.io/projected/aa515af9-3fdb-403d-bd09-837323c156f6-kube-api-access-6ffgv\") pod \"kuadrant-console-plugin-598f86468b-98s8s\" (UID: \"aa515af9-3fdb-403d-bd09-837323c156f6\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-98s8s"
Sep 05 19:53:17.810183 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:17.810148    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa515af9-3fdb-403d-bd09-837323c156f6-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-98s8s\" (UID: \"aa515af9-3fdb-403d-bd09-837323c156f6\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-98s8s"
Sep 05 19:53:17.810346 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:17.810194    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ffgv\" (UniqueName: \"kubernetes.io/projected/aa515af9-3fdb-403d-bd09-837323c156f6-kube-api-access-6ffgv\") pod \"kuadrant-console-plugin-598f86468b-98s8s\" (UID: \"aa515af9-3fdb-403d-bd09-837323c156f6\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-98s8s"
Sep 05 19:53:17.810346 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:17.810268    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/aa515af9-3fdb-403d-bd09-837323c156f6-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-98s8s\" (UID: \"aa515af9-3fdb-403d-bd09-837323c156f6\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-98s8s"
Sep 05 19:53:17.810346 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:17.810282    2583 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 05 19:53:17.810447 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:17.810363    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa515af9-3fdb-403d-bd09-837323c156f6-plugin-serving-cert podName:aa515af9-3fdb-403d-bd09-837323c156f6 nodeName:}" failed. No retries permitted until 2026-09-05 19:53:18.310341386 +0000 UTC m=+597.339973279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/aa515af9-3fdb-403d-bd09-837323c156f6-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-98s8s" (UID: "aa515af9-3fdb-403d-bd09-837323c156f6") : secret "plugin-serving-cert" not found
Sep 05 19:53:17.810899 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:17.810879    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/aa515af9-3fdb-403d-bd09-837323c156f6-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-98s8s\" (UID: \"aa515af9-3fdb-403d-bd09-837323c156f6\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-98s8s"
Sep 05 19:53:17.822147 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:17.822093    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ffgv\" (UniqueName: \"kubernetes.io/projected/aa515af9-3fdb-403d-bd09-837323c156f6-kube-api-access-6ffgv\") pod \"kuadrant-console-plugin-598f86468b-98s8s\" (UID: \"aa515af9-3fdb-403d-bd09-837323c156f6\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-98s8s"
Sep 05 19:53:18.314589 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:18.314548    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa515af9-3fdb-403d-bd09-837323c156f6-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-98s8s\" (UID: \"aa515af9-3fdb-403d-bd09-837323c156f6\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-98s8s"
Sep 05 19:53:18.316971 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:18.316941    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa515af9-3fdb-403d-bd09-837323c156f6-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-98s8s\" (UID: \"aa515af9-3fdb-403d-bd09-837323c156f6\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-98s8s"
Sep 05 19:53:18.538780 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:18.538743    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-98s8s"
Sep 05 19:53:18.806698 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:18.806672    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-98s8s"]
Sep 05 19:53:18.809221 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:53:18.809194    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa515af9_3fdb_403d_bd09_837323c156f6.slice/crio-0ef699344fbb6966af634234b434d576c23915ddf858f9a900520210f30642f8 WatchSource:0}: Error finding container 0ef699344fbb6966af634234b434d576c23915ddf858f9a900520210f30642f8: Status 404 returned error can't find the container with id 0ef699344fbb6966af634234b434d576c23915ddf858f9a900520210f30642f8
Sep 05 19:53:18.901063 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:18.900984    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vpk9d"
Sep 05 19:53:18.959055 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:18.959020    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-98s8s" event={"ID":"aa515af9-3fdb-403d-bd09-837323c156f6","Type":"ContainerStarted","Data":"0ef699344fbb6966af634234b434d576c23915ddf858f9a900520210f30642f8"}
Sep 05 19:53:21.383776 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:21.383750    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf9tj_f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b/ovn-acl-logging/0.log"
Sep 05 19:53:21.384188 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:21.384022    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf9tj_f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b/ovn-acl-logging/0.log"
Sep 05 19:53:26.945643 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:26.945606    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9"
Sep 05 19:53:27.970583 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:27.970541    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9"]
Sep 05 19:53:27.974399 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:27.974377    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9"
Sep 05 19:53:27.989791 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:27.989766    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9"]
Sep 05 19:53:28.014281 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:28.014255    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-98s8s" event={"ID":"aa515af9-3fdb-403d-bd09-837323c156f6","Type":"ContainerStarted","Data":"3984fd3e1e13575159f951b4050d2d8d45af550a4de93eded65e45a5f3b1703d"}
Sep 05 19:53:28.030099 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:28.030056    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-98s8s" podStartSLOduration=2.6192874 podStartE2EDuration="11.030042031s" totalImagesPullingTime="8.410754631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:53:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:53:28.02964866 +0000 UTC m=+607.059280584" watchObservedRunningTime="2026-09-05 19:53:28.030042031 +0000 UTC m=+607.059674032"
Sep 05 19:53:28.099943 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:28.099913    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec9abff9-43a7-4edd-b2ce-81e736365a27-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-9n6c9\" (UID: \"ec9abff9-43a7-4edd-b2ce-81e736365a27\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9"
Sep 05 19:53:28.100168 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:28.099949    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vngzc\" (UniqueName: \"kubernetes.io/projected/ec9abff9-43a7-4edd-b2ce-81e736365a27-kube-api-access-vngzc\") pod \"kuadrant-operator-controller-manager-96db98c47-9n6c9\" (UID: \"ec9abff9-43a7-4edd-b2ce-81e736365a27\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9"
Sep 05 19:53:28.200928 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:28.200889    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec9abff9-43a7-4edd-b2ce-81e736365a27-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-9n6c9\" (UID: \"ec9abff9-43a7-4edd-b2ce-81e736365a27\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9"
Sep 05 19:53:28.200928 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:28.200930    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vngzc\" (UniqueName: \"kubernetes.io/projected/ec9abff9-43a7-4edd-b2ce-81e736365a27-kube-api-access-vngzc\") pod \"kuadrant-operator-controller-manager-96db98c47-9n6c9\" (UID: \"ec9abff9-43a7-4edd-b2ce-81e736365a27\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9"
Sep 05 19:53:28.201278 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:28.201256    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec9abff9-43a7-4edd-b2ce-81e736365a27-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-9n6c9\" (UID: \"ec9abff9-43a7-4edd-b2ce-81e736365a27\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9"
Sep 05 19:53:28.209296 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:28.209272    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vngzc\" (UniqueName: \"kubernetes.io/projected/ec9abff9-43a7-4edd-b2ce-81e736365a27-kube-api-access-vngzc\") pod \"kuadrant-operator-controller-manager-96db98c47-9n6c9\" (UID: \"ec9abff9-43a7-4edd-b2ce-81e736365a27\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9"
Sep 05 19:53:28.284445 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:28.284363    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9"
Sep 05 19:53:28.563747 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:28.563637    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9"]
Sep 05 19:53:28.568726 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:53:28.568700    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9abff9_43a7_4edd_b2ce_81e736365a27.slice/crio-f5ba4742e0eb599646dc27bfe4c72de732ba75e7270b4982c168587714cb85d0 WatchSource:0}: Error finding container f5ba4742e0eb599646dc27bfe4c72de732ba75e7270b4982c168587714cb85d0: Status 404 returned error can't find the container with id f5ba4742e0eb599646dc27bfe4c72de732ba75e7270b4982c168587714cb85d0
Sep 05 19:53:28.976761 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:28.976718    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9"]
Sep 05 19:53:28.977249 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:28.976958    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9" podUID="a01ebc67-0f15-479d-b91f-02d146697eea" containerName="manager" containerID="cri-o://759a782e8d8c0138d9743affbc5e7469101ad3da25722c56369e30e7d152effa" gracePeriod=2
Sep 05 19:53:28.981688 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:28.981660    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9"]
Sep 05 19:53:28.995682 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:28.995657    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9"]
Sep 05 19:53:29.002944 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.002915    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-l58p5"]
Sep 05 19:53:29.003475 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.003455    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a01ebc67-0f15-479d-b91f-02d146697eea" containerName="manager"
Sep 05 19:53:29.003475 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.003475    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a01ebc67-0f15-479d-b91f-02d146697eea" containerName="manager"
Sep 05 19:53:29.003646 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.003556    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a01ebc67-0f15-479d-b91f-02d146697eea" containerName="manager"
Sep 05 19:53:29.006879 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.006858    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-l58p5"
Sep 05 19:53:29.007419 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.007398    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9"]
Sep 05 19:53:29.008736 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:29.008700    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-xqxt9\" is forbidden: User \"system:node:ip-10-0-142-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-208.ec2.internal' and this object" podUID="a01ebc67-0f15-479d-b91f-02d146697eea" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9"
Sep 05 19:53:29.021205 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.020869    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-l58p5"]
Sep 05 19:53:29.024945 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.024749    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-w8cvv"]
Sep 05 19:53:29.025340 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.025315    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec9abff9-43a7-4edd-b2ce-81e736365a27" containerName="manager"
Sep 05 19:53:29.025340 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.025341    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec9abff9-43a7-4edd-b2ce-81e736365a27" containerName="manager"
Sep 05 19:53:29.025475 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.025431    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec9abff9-43a7-4edd-b2ce-81e736365a27" containerName="manager"
Sep 05 19:53:29.027002 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:29.026979    2583 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9" container="manager"
Sep 05 19:53:29.029014 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.028997    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7n7z"]
Sep 05 19:53:29.029258 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.029171    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7n7z" podUID="a067b15a-1fd3-4969-93ff-49a56a5e376c" containerName="manager" containerID="cri-o://1b3d20eaee778a0a249781698514a2a87779b6b511046ad8b06175140cb4fbd6" gracePeriod=2
Sep 05 19:53:29.029348 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.029289    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-w8cvv"
Sep 05 19:53:29.031186 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.031162    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7n7z"]
Sep 05 19:53:29.042223 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.042199    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-w8cvv"]
Sep 05 19:53:29.052022 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.052000    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xfb64"]
Sep 05 19:53:29.052335 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.052322    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a067b15a-1fd3-4969-93ff-49a56a5e376c" containerName="manager"
Sep 05 19:53:29.052396 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.052337    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a067b15a-1fd3-4969-93ff-49a56a5e376c" containerName="manager"
Sep 05 19:53:29.052452 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.052419    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a067b15a-1fd3-4969-93ff-49a56a5e376c" containerName="manager"
Sep 05 19:53:29.055496 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.055480    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xfb64"
Sep 05 19:53:29.057809 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:29.057779    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-v7n7z\" is forbidden: User \"system:node:ip-10-0-142-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-208.ec2.internal' and this object" podUID="a067b15a-1fd3-4969-93ff-49a56a5e376c" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7n7z"
Sep 05 19:53:29.059410 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:29.059371    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-xqxt9\" is forbidden: User \"system:node:ip-10-0-142-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-208.ec2.internal' and this object" podUID="a01ebc67-0f15-479d-b91f-02d146697eea" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9"
Sep 05 19:53:29.060885 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:29.060856    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-9n6c9\" is forbidden: User \"system:node:ip-10-0-142-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-208.ec2.internal' and this object" podUID="ec9abff9-43a7-4edd-b2ce-81e736365a27" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9"
Sep 05 19:53:29.064995 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.064957    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xfb64"]
Sep 05 19:53:29.079301 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:29.079266    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-v7n7z\" is forbidden: User \"system:node:ip-10-0-142-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-208.ec2.internal' and this object" podUID="a067b15a-1fd3-4969-93ff-49a56a5e376c" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7n7z"
Sep 05 19:53:29.080682 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:29.080640    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-xqxt9\" is forbidden: User \"system:node:ip-10-0-142-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-208.ec2.internal' and this object" podUID="a01ebc67-0f15-479d-b91f-02d146697eea" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9"
Sep 05 19:53:29.081956 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:29.081927    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-9n6c9\" is forbidden: User \"system:node:ip-10-0-142-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-208.ec2.internal' and this object" podUID="ec9abff9-43a7-4edd-b2ce-81e736365a27" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9"
Sep 05 19:53:29.108985 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.108962    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqb9l\" (UniqueName: \"kubernetes.io/projected/f91d8355-8880-4180-b0c1-112df3ee0cd6-kube-api-access-gqb9l\") pod \"kuadrant-operator-controller-manager-96db98c47-w8cvv\" (UID: \"f91d8355-8880-4180-b0c1-112df3ee0cd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-w8cvv"
Sep 05 19:53:29.109061 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.109005    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7013db1-e2cf-4c0c-8116-be49d108151d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-l58p5\" (UID: \"a7013db1-e2cf-4c0c-8116-be49d108151d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-l58p5"
Sep 05 19:53:29.109124 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.109085    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f91d8355-8880-4180-b0c1-112df3ee0cd6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-w8cvv\" (UID: \"f91d8355-8880-4180-b0c1-112df3ee0cd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-w8cvv"
Sep 05 19:53:29.109159 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.109145    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkb82\" (UniqueName: \"kubernetes.io/projected/a7013db1-e2cf-4c0c-8116-be49d108151d-kube-api-access-jkb82\") pod \"kuadrant-operator-controller-manager-58fc56757c-l58p5\" (UID: \"a7013db1-e2cf-4c0c-8116-be49d108151d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-l58p5"
Sep 05 19:53:29.109195 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.109175    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w849c\" (UniqueName: \"kubernetes.io/projected/02a54913-6879-42b0-96a4-e1584b934e40-kube-api-access-w849c\") pod \"limitador-operator-controller-manager-6c549b6d85-xfb64\" (UID: \"02a54913-6879-42b0-96a4-e1584b934e40\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xfb64"
Sep 05 19:53:29.210860 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.210705    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqb9l\" (UniqueName: \"kubernetes.io/projected/f91d8355-8880-4180-b0c1-112df3ee0cd6-kube-api-access-gqb9l\") pod \"kuadrant-operator-controller-manager-96db98c47-w8cvv\" (UID: \"f91d8355-8880-4180-b0c1-112df3ee0cd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-w8cvv"
Sep 05 19:53:29.210860 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.210756    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7013db1-e2cf-4c0c-8116-be49d108151d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-l58p5\" (UID: \"a7013db1-e2cf-4c0c-8116-be49d108151d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-l58p5"
Sep 05 19:53:29.210860 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.210818    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f91d8355-8880-4180-b0c1-112df3ee0cd6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-w8cvv\" (UID: \"f91d8355-8880-4180-b0c1-112df3ee0cd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-w8cvv"
Sep 05 19:53:29.210860 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.210852    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkb82\" (UniqueName: \"kubernetes.io/projected/a7013db1-e2cf-4c0c-8116-be49d108151d-kube-api-access-jkb82\") pod \"kuadrant-operator-controller-manager-58fc56757c-l58p5\" (UID: \"a7013db1-e2cf-4c0c-8116-be49d108151d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-l58p5"
Sep 05 19:53:29.211163 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.210880    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w849c\" (UniqueName: \"kubernetes.io/projected/02a54913-6879-42b0-96a4-e1584b934e40-kube-api-access-w849c\") pod \"limitador-operator-controller-manager-6c549b6d85-xfb64\" (UID: \"02a54913-6879-42b0-96a4-e1584b934e40\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xfb64"
Sep 05 19:53:29.211420 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.211390    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7013db1-e2cf-4c0c-8116-be49d108151d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-l58p5\" (UID: \"a7013db1-e2cf-4c0c-8116-be49d108151d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-l58p5"
Sep 05 19:53:29.211513 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.211415    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f91d8355-8880-4180-b0c1-112df3ee0cd6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-w8cvv\" (UID: \"f91d8355-8880-4180-b0c1-112df3ee0cd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-w8cvv"
Sep 05 19:53:29.221322 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.221258    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqb9l\" (UniqueName: \"kubernetes.io/projected/f91d8355-8880-4180-b0c1-112df3ee0cd6-kube-api-access-gqb9l\") pod \"kuadrant-operator-controller-manager-96db98c47-w8cvv\" (UID: \"f91d8355-8880-4180-b0c1-112df3ee0cd6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-w8cvv"
Sep 05 19:53:29.221787 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.221765    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkb82\" (UniqueName: \"kubernetes.io/projected/a7013db1-e2cf-4c0c-8116-be49d108151d-kube-api-access-jkb82\") pod \"kuadrant-operator-controller-manager-58fc56757c-l58p5\" (UID: \"a7013db1-e2cf-4c0c-8116-be49d108151d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-l58p5"
Sep 05 19:53:29.221868 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.221789    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w849c\" (UniqueName: \"kubernetes.io/projected/02a54913-6879-42b0-96a4-e1584b934e40-kube-api-access-w849c\") pod \"limitador-operator-controller-manager-6c549b6d85-xfb64\" (UID: \"02a54913-6879-42b0-96a4-e1584b934e40\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xfb64"
Sep 05 19:53:29.258536 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.258515    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9"
Sep 05 19:53:29.260377 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:29.260341    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-xqxt9\" is forbidden: User \"system:node:ip-10-0-142-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-208.ec2.internal' and this object" podUID="a01ebc67-0f15-479d-b91f-02d146697eea" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9"
Sep 05 19:53:29.262039 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:29.262008    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-9n6c9\" is forbidden: User \"system:node:ip-10-0-142-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-208.ec2.internal' and this object" podUID="ec9abff9-43a7-4edd-b2ce-81e736365a27" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9"
Sep 05 19:53:29.263732 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:29.263707    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-v7n7z\" is forbidden: User \"system:node:ip-10-0-142-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-208.ec2.internal' and this object" podUID="a067b15a-1fd3-4969-93ff-49a56a5e376c" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7n7z"
Sep 05 19:53:29.265169 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.265151    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7n7z"
Sep 05 19:53:29.267140 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:29.267090    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-xqxt9\" is forbidden: User \"system:node:ip-10-0-142-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-208.ec2.internal' and this object" podUID="a01ebc67-0f15-479d-b91f-02d146697eea" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9"
Sep 05 19:53:29.268641 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:29.268616    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-9n6c9\" is forbidden: User \"system:node:ip-10-0-142-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-208.ec2.internal' and this object" podUID="ec9abff9-43a7-4edd-b2ce-81e736365a27" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9"
Sep 05 19:53:29.269941 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:29.269919    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-v7n7z\" is forbidden: User \"system:node:ip-10-0-142-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-208.ec2.internal' and this object" podUID="a067b15a-1fd3-4969-93ff-49a56a5e376c" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7n7z"
Sep 05 19:53:29.312399 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.312376    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqhfj\" (UniqueName: \"kubernetes.io/projected/a01ebc67-0f15-479d-b91f-02d146697eea-kube-api-access-bqhfj\") pod \"a01ebc67-0f15-479d-b91f-02d146697eea\" (UID: \"a01ebc67-0f15-479d-b91f-02d146697eea\") "
Sep 05 19:53:29.312504 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.312418    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a01ebc67-0f15-479d-b91f-02d146697eea-extensions-socket-volume\") pod \"a01ebc67-0f15-479d-b91f-02d146697eea\" (UID: \"a01ebc67-0f15-479d-b91f-02d146697eea\") "
Sep 05 19:53:29.312839 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.312816    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01ebc67-0f15-479d-b91f-02d146697eea-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a01ebc67-0f15-479d-b91f-02d146697eea" (UID: "a01ebc67-0f15-479d-b91f-02d146697eea"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:53:29.314638 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.314616    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01ebc67-0f15-479d-b91f-02d146697eea-kube-api-access-bqhfj" (OuterVolumeSpecName: "kube-api-access-bqhfj") pod "a01ebc67-0f15-479d-b91f-02d146697eea" (UID: "a01ebc67-0f15-479d-b91f-02d146697eea"). InnerVolumeSpecName "kube-api-access-bqhfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:53:29.386497 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.386467    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-l58p5"
Sep 05 19:53:29.395425 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.395399    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-w8cvv"
Sep 05 19:53:29.401503 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.401444    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xfb64"
Sep 05 19:53:29.413587 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.413534    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5xc5\" (UniqueName: \"kubernetes.io/projected/a067b15a-1fd3-4969-93ff-49a56a5e376c-kube-api-access-h5xc5\") pod \"a067b15a-1fd3-4969-93ff-49a56a5e376c\" (UID: \"a067b15a-1fd3-4969-93ff-49a56a5e376c\") "
Sep 05 19:53:29.413832 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.413796    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqhfj\" (UniqueName: \"kubernetes.io/projected/a01ebc67-0f15-479d-b91f-02d146697eea-kube-api-access-bqhfj\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:53:29.413832 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.413814    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a01ebc67-0f15-479d-b91f-02d146697eea-extensions-socket-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:53:29.415956 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.415930    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a067b15a-1fd3-4969-93ff-49a56a5e376c-kube-api-access-h5xc5" (OuterVolumeSpecName: "kube-api-access-h5xc5") pod "a067b15a-1fd3-4969-93ff-49a56a5e376c" (UID: "a067b15a-1fd3-4969-93ff-49a56a5e376c"). InnerVolumeSpecName "kube-api-access-h5xc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:53:29.440197 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.440165    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a01ebc67-0f15-479d-b91f-02d146697eea" path="/var/lib/kubelet/pods/a01ebc67-0f15-479d-b91f-02d146697eea/volumes"
Sep 05 19:53:29.441072 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.440835    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a067b15a-1fd3-4969-93ff-49a56a5e376c" path="/var/lib/kubelet/pods/a067b15a-1fd3-4969-93ff-49a56a5e376c/volumes"
Sep 05 19:53:29.514719 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.514647    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5xc5\" (UniqueName: \"kubernetes.io/projected/a067b15a-1fd3-4969-93ff-49a56a5e376c-kube-api-access-h5xc5\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:53:29.668743 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.668716    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-l58p5"]
Sep 05 19:53:29.670756 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:53:29.670729    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7013db1_e2cf_4c0c_8116_be49d108151d.slice/crio-df59a31481247ee0c99c443381260159bcf706ea2b3a931bd806a6fb1f152652 WatchSource:0}: Error finding container df59a31481247ee0c99c443381260159bcf706ea2b3a931bd806a6fb1f152652: Status 404 returned error can't find the container with id df59a31481247ee0c99c443381260159bcf706ea2b3a931bd806a6fb1f152652
Sep 05 19:53:29.690309 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.690280    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-w8cvv"]
Sep 05 19:53:29.692417 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:53:29.692381    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91d8355_8880_4180_b0c1_112df3ee0cd6.slice/crio-4d1843402d560ed750c86e9c826d6e46c429bcd68b68c9142538f31708a01712 WatchSource:0}: Error finding container 4d1843402d560ed750c86e9c826d6e46c429bcd68b68c9142538f31708a01712: Status 404 returned error can't find the container with id 4d1843402d560ed750c86e9c826d6e46c429bcd68b68c9142538f31708a01712
Sep 05 19:53:29.693550 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:29.693529    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xfb64"]
Sep 05 19:53:29.694415 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:53:29.694382    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a54913_6879_42b0_96a4_e1584b934e40.slice/crio-aaf0bde5ee07ac72efc92e118e57209f463487bcaa04d0ecaace7eecf9feae29 WatchSource:0}: Error finding container aaf0bde5ee07ac72efc92e118e57209f463487bcaa04d0ecaace7eecf9feae29: Status 404 returned error can't find the container with id aaf0bde5ee07ac72efc92e118e57209f463487bcaa04d0ecaace7eecf9feae29
Sep 05 19:53:30.033993 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.033955    2583 generic.go:356] "Generic (PLEG): container finished" podID="a01ebc67-0f15-479d-b91f-02d146697eea" containerID="759a782e8d8c0138d9743affbc5e7469101ad3da25722c56369e30e7d152effa" exitCode=0
Sep 05 19:53:30.034424 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.034016    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-xqxt9"
Sep 05 19:53:30.034424 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.034028    2583 scope.go:117] "RemoveContainer" containerID="759a782e8d8c0138d9743affbc5e7469101ad3da25722c56369e30e7d152effa"
Sep 05 19:53:30.036187 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.036148    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xfb64" event={"ID":"02a54913-6879-42b0-96a4-e1584b934e40","Type":"ContainerStarted","Data":"d2b779a8c4b4d51e5a4453dc544f97bb05d1d2a431bb4e2dfa0a39a13094f77f"}
Sep 05 19:53:30.036422 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.036402    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xfb64"
Sep 05 19:53:30.036585 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.036561    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xfb64" event={"ID":"02a54913-6879-42b0-96a4-e1584b934e40","Type":"ContainerStarted","Data":"aaf0bde5ee07ac72efc92e118e57209f463487bcaa04d0ecaace7eecf9feae29"}
Sep 05 19:53:30.038171 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.038142    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-l58p5" event={"ID":"a7013db1-e2cf-4c0c-8116-be49d108151d","Type":"ContainerStarted","Data":"eff1a25241539d89e881a9c641e139a3b0341cdc08531e9f546e6e9c6ffa6d36"}
Sep 05 19:53:30.038280 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.038199    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-l58p5" event={"ID":"a7013db1-e2cf-4c0c-8116-be49d108151d","Type":"ContainerStarted","Data":"df59a31481247ee0c99c443381260159bcf706ea2b3a931bd806a6fb1f152652"}
Sep 05 19:53:30.038280 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.038228    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-l58p5"
Sep 05 19:53:30.040060 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.040026    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9" podUID="ec9abff9-43a7-4edd-b2ce-81e736365a27" containerName="manager" containerID="cri-o://8256d82f6a60351a23ef39f1f56c9a654c329894ddce71931758442932bd63a7" gracePeriod=2
Sep 05 19:53:30.041434 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.041414    2583 generic.go:356] "Generic (PLEG): container finished" podID="a067b15a-1fd3-4969-93ff-49a56a5e376c" containerID="1b3d20eaee778a0a249781698514a2a87779b6b511046ad8b06175140cb4fbd6" exitCode=0
Sep 05 19:53:30.041527 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.041471    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-v7n7z"
Sep 05 19:53:30.043825 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.043803    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-w8cvv" event={"ID":"f91d8355-8880-4180-b0c1-112df3ee0cd6","Type":"ContainerStarted","Data":"f57495eb63c8c6eb9f0f20c2bb9668876e90be25e76e650a51b5719870cfe2c3"}
Sep 05 19:53:30.044006 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.043986    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-w8cvv" event={"ID":"f91d8355-8880-4180-b0c1-112df3ee0cd6","Type":"ContainerStarted","Data":"4d1843402d560ed750c86e9c826d6e46c429bcd68b68c9142538f31708a01712"}
Sep 05 19:53:30.044248 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.044234    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-w8cvv"
Sep 05 19:53:30.045006 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.044840    2583 scope.go:117] "RemoveContainer" containerID="759a782e8d8c0138d9743affbc5e7469101ad3da25722c56369e30e7d152effa"
Sep 05 19:53:30.045208 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:30.045184    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"759a782e8d8c0138d9743affbc5e7469101ad3da25722c56369e30e7d152effa\": container with ID starting with 759a782e8d8c0138d9743affbc5e7469101ad3da25722c56369e30e7d152effa not found: ID does not exist" containerID="759a782e8d8c0138d9743affbc5e7469101ad3da25722c56369e30e7d152effa"
Sep 05 19:53:30.045302 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.045219    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"759a782e8d8c0138d9743affbc5e7469101ad3da25722c56369e30e7d152effa"} err="failed to get container status \"759a782e8d8c0138d9743affbc5e7469101ad3da25722c56369e30e7d152effa\": rpc error: code = NotFound desc = could not find container \"759a782e8d8c0138d9743affbc5e7469101ad3da25722c56369e30e7d152effa\": container with ID starting with 759a782e8d8c0138d9743affbc5e7469101ad3da25722c56369e30e7d152effa not found: ID does not exist"
Sep 05 19:53:30.045302 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.045240    2583 scope.go:117] "RemoveContainer" containerID="1b3d20eaee778a0a249781698514a2a87779b6b511046ad8b06175140cb4fbd6"
Sep 05 19:53:30.053618 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.053602    2583 scope.go:117] "RemoveContainer" containerID="1b3d20eaee778a0a249781698514a2a87779b6b511046ad8b06175140cb4fbd6"
Sep 05 19:53:30.053874 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:30.053857    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3d20eaee778a0a249781698514a2a87779b6b511046ad8b06175140cb4fbd6\": container with ID starting with 1b3d20eaee778a0a249781698514a2a87779b6b511046ad8b06175140cb4fbd6 not found: ID does not exist" containerID="1b3d20eaee778a0a249781698514a2a87779b6b511046ad8b06175140cb4fbd6"
Sep 05 19:53:30.053935 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.053879    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3d20eaee778a0a249781698514a2a87779b6b511046ad8b06175140cb4fbd6"} err="failed to get container status \"1b3d20eaee778a0a249781698514a2a87779b6b511046ad8b06175140cb4fbd6\": rpc error: code = NotFound desc = could not find container \"1b3d20eaee778a0a249781698514a2a87779b6b511046ad8b06175140cb4fbd6\": container with ID starting with 1b3d20eaee778a0a249781698514a2a87779b6b511046ad8b06175140cb4fbd6 not found: ID does not exist"
Sep 05 19:53:30.059261 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.059224    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xfb64" podStartSLOduration=1.059212519 podStartE2EDuration="1.059212519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:53:30.057679986 +0000 UTC m=+609.087311882" watchObservedRunningTime="2026-09-05 19:53:30.059212519 +0000 UTC m=+609.088844426"
Sep 05 19:53:30.085669 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.085605    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-w8cvv" podStartSLOduration=2.08558952 podStartE2EDuration="2.08558952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:53:30.08515215 +0000 UTC m=+609.114784055" watchObservedRunningTime="2026-09-05 19:53:30.08558952 +0000 UTC m=+609.115221426"
Sep 05 19:53:30.109014 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.108960    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-l58p5" podStartSLOduration=2.108947093 podStartE2EDuration="2.108947093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:53:30.106428043 +0000 UTC m=+609.136059950" watchObservedRunningTime="2026-09-05 19:53:30.108947093 +0000 UTC m=+609.138578998"
Sep 05 19:53:30.277544 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.277520    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9"
Sep 05 19:53:30.279842 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:30.279817    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-9n6c9\" is forbidden: User \"system:node:ip-10-0-142-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-208.ec2.internal' and this object" podUID="ec9abff9-43a7-4edd-b2ce-81e736365a27" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9"
Sep 05 19:53:30.422592 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.422554    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec9abff9-43a7-4edd-b2ce-81e736365a27-extensions-socket-volume\") pod \"ec9abff9-43a7-4edd-b2ce-81e736365a27\" (UID: \"ec9abff9-43a7-4edd-b2ce-81e736365a27\") "
Sep 05 19:53:30.422776 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.422606    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vngzc\" (UniqueName: \"kubernetes.io/projected/ec9abff9-43a7-4edd-b2ce-81e736365a27-kube-api-access-vngzc\") pod \"ec9abff9-43a7-4edd-b2ce-81e736365a27\" (UID: \"ec9abff9-43a7-4edd-b2ce-81e736365a27\") "
Sep 05 19:53:30.422916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.422889    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9abff9-43a7-4edd-b2ce-81e736365a27-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ec9abff9-43a7-4edd-b2ce-81e736365a27" (UID: "ec9abff9-43a7-4edd-b2ce-81e736365a27"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:53:30.424710 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.424684    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9abff9-43a7-4edd-b2ce-81e736365a27-kube-api-access-vngzc" (OuterVolumeSpecName: "kube-api-access-vngzc") pod "ec9abff9-43a7-4edd-b2ce-81e736365a27" (UID: "ec9abff9-43a7-4edd-b2ce-81e736365a27"). InnerVolumeSpecName "kube-api-access-vngzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:53:30.523171 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.523139    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vngzc\" (UniqueName: \"kubernetes.io/projected/ec9abff9-43a7-4edd-b2ce-81e736365a27-kube-api-access-vngzc\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:53:30.523171 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:30.523169    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec9abff9-43a7-4edd-b2ce-81e736365a27-extensions-socket-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:53:31.050811 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:31.050772    2583 generic.go:356] "Generic (PLEG): container finished" podID="ec9abff9-43a7-4edd-b2ce-81e736365a27" containerID="8256d82f6a60351a23ef39f1f56c9a654c329894ddce71931758442932bd63a7" exitCode=0
Sep 05 19:53:31.051248 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:31.050814    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9"
Sep 05 19:53:31.051248 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:31.050868    2583 scope.go:117] "RemoveContainer" containerID="8256d82f6a60351a23ef39f1f56c9a654c329894ddce71931758442932bd63a7"
Sep 05 19:53:31.052945 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:31.052920    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-9n6c9\" is forbidden: User \"system:node:ip-10-0-142-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-208.ec2.internal' and this object" podUID="ec9abff9-43a7-4edd-b2ce-81e736365a27" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9"
Sep 05 19:53:31.059989 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:31.059973    2583 scope.go:117] "RemoveContainer" containerID="8256d82f6a60351a23ef39f1f56c9a654c329894ddce71931758442932bd63a7"
Sep 05 19:53:31.060271 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:31.060249    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8256d82f6a60351a23ef39f1f56c9a654c329894ddce71931758442932bd63a7\": container with ID starting with 8256d82f6a60351a23ef39f1f56c9a654c329894ddce71931758442932bd63a7 not found: ID does not exist" containerID="8256d82f6a60351a23ef39f1f56c9a654c329894ddce71931758442932bd63a7"
Sep 05 19:53:31.060355 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:31.060277    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8256d82f6a60351a23ef39f1f56c9a654c329894ddce71931758442932bd63a7"} err="failed to get container status \"8256d82f6a60351a23ef39f1f56c9a654c329894ddce71931758442932bd63a7\": rpc error: code = NotFound desc = could not find container \"8256d82f6a60351a23ef39f1f56c9a654c329894ddce71931758442932bd63a7\": container with ID starting with 8256d82f6a60351a23ef39f1f56c9a654c329894ddce71931758442932bd63a7 not found: ID does not exist"
Sep 05 19:53:31.060861 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:31.060839    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-9n6c9\" is forbidden: User \"system:node:ip-10-0-142-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-208.ec2.internal' and this object" podUID="ec9abff9-43a7-4edd-b2ce-81e736365a27" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9"
Sep 05 19:53:31.433358 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:31.433319    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-9n6c9\" is forbidden: User \"system:node:ip-10-0-142-208.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-208.ec2.internal' and this object" podUID="ec9abff9-43a7-4edd-b2ce-81e736365a27" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-9n6c9"
Sep 05 19:53:31.434686 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:31.434666    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec9abff9-43a7-4edd-b2ce-81e736365a27" path="/var/lib/kubelet/pods/ec9abff9-43a7-4edd-b2ce-81e736365a27/volumes"
Sep 05 19:53:41.055857 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:41.055825    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-w8cvv"
Sep 05 19:53:41.056321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:41.055997    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-l58p5"
Sep 05 19:53:41.142584 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:41.142552    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-l58p5"]
Sep 05 19:53:41.142805 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:41.142781    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-l58p5" podUID="a7013db1-e2cf-4c0c-8116-be49d108151d" containerName="manager" containerID="cri-o://eff1a25241539d89e881a9c641e139a3b0341cdc08531e9f546e6e9c6ffa6d36" gracePeriod=10
Sep 05 19:53:41.390629 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:41.390602    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-l58p5"
Sep 05 19:53:41.404872 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:41.404847    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-kg8m8"]
Sep 05 19:53:41.405232 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:41.405220    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7013db1-e2cf-4c0c-8116-be49d108151d" containerName="manager"
Sep 05 19:53:41.405278 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:41.405233    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7013db1-e2cf-4c0c-8116-be49d108151d" containerName="manager"
Sep 05 19:53:41.405313 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:41.405294    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7013db1-e2cf-4c0c-8116-be49d108151d" containerName="manager"
Sep 05 19:53:41.408418 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:41.408398    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-kg8m8"
Sep 05 19:53:41.437931 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:41.437907    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-kg8m8"]
Sep 05 19:53:41.520401 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:41.520357    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7013db1-e2cf-4c0c-8116-be49d108151d-extensions-socket-volume\") pod \"a7013db1-e2cf-4c0c-8116-be49d108151d\" (UID: \"a7013db1-e2cf-4c0c-8116-be49d108151d\") "
Sep 05 19:53:41.520562 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:41.520423    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkb82\" (UniqueName: \"kubernetes.io/projected/a7013db1-e2cf-4c0c-8116-be49d108151d-kube-api-access-jkb82\") pod \"a7013db1-e2cf-4c0c-8116-be49d108151d\" (UID: \"a7013db1-e2cf-4c0c-8116-be49d108151d\") "
Sep 05 19:53:41.520624 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:41.520590    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3259161d-2c56-4705-aca8-ff45669a1915-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-kg8m8\" (UID: \"3259161d-2c56-4705-aca8-ff45669a1915\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-kg8m8"
Sep 05 19:53:41.520815 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:41.520784    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tcdd\" (UniqueName: \"kubernetes.io/projected/3259161d-2c56-4705-aca8-ff45669a1915-kube-api-access-4tcdd\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-kg8m8\" (UID: \"3259161d-2c56-4705-aca8-ff45669a1915\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-kg8m8"
Sep 05 19:53:41.521057 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:41.520812    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7013db1-e2cf-4c0c-8116-be49d108151d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a7013db1-e2cf-4c0c-8116-be49d108151d" (UID: "a7013db1-e2cf-4c0c-8116-be49d108151d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:53:41.522827 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:41.522797    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7013db1-e2cf-4c0c-8116-be49d108151d-kube-api-access-jkb82" (OuterVolumeSpecName: "kube-api-access-jkb82") pod "a7013db1-e2cf-4c0c-8116-be49d108151d" (UID: "a7013db1-e2cf-4c0c-8116-be49d108151d"). InnerVolumeSpecName "kube-api-access-jkb82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:53:41.622003 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:41.621965    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tcdd\" (UniqueName: \"kubernetes.io/projected/3259161d-2c56-4705-aca8-ff45669a1915-kube-api-access-4tcdd\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-kg8m8\" (UID: \"3259161d-2c56-4705-aca8-ff45669a1915\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-kg8m8"
Sep 05 19:53:41.622199 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:41.622031    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3259161d-2c56-4705-aca8-ff45669a1915-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-kg8m8\" (UID: \"3259161d-2c56-4705-aca8-ff45669a1915\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-kg8m8"
Sep 05 19:53:41.622199 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:41.622078    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7013db1-e2cf-4c0c-8116-be49d108151d-extensions-socket-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:53:41.622199 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:41.622092    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkb82\" (UniqueName: \"kubernetes.io/projected/a7013db1-e2cf-4c0c-8116-be49d108151d-kube-api-access-jkb82\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:53:41.622432 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:41.622409    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3259161d-2c56-4705-aca8-ff45669a1915-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-kg8m8\" (UID: \"3259161d-2c56-4705-aca8-ff45669a1915\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-kg8m8"
Sep 05 19:53:41.630838 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:41.630816    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tcdd\" (UniqueName: \"kubernetes.io/projected/3259161d-2c56-4705-aca8-ff45669a1915-kube-api-access-4tcdd\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-kg8m8\" (UID: \"3259161d-2c56-4705-aca8-ff45669a1915\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-kg8m8"
Sep 05 19:53:41.718103 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:41.718075    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-kg8m8"
Sep 05 19:53:42.003386 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:42.003360    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-kg8m8"]
Sep 05 19:53:42.005297 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:53:42.005274    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3259161d_2c56_4705_aca8_ff45669a1915.slice/crio-d06aab4a8d88959e0fcf09b4dc631424ef8f3985331b0befea30c34ace168d89 WatchSource:0}: Error finding container d06aab4a8d88959e0fcf09b4dc631424ef8f3985331b0befea30c34ace168d89: Status 404 returned error can't find the container with id d06aab4a8d88959e0fcf09b4dc631424ef8f3985331b0befea30c34ace168d89
Sep 05 19:53:42.062972 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:42.062951    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-xfb64"
Sep 05 19:53:42.120621 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:42.120583    2583 generic.go:356] "Generic (PLEG): container finished" podID="a7013db1-e2cf-4c0c-8116-be49d108151d" containerID="eff1a25241539d89e881a9c641e139a3b0341cdc08531e9f546e6e9c6ffa6d36" exitCode=0
Sep 05 19:53:42.120788 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:42.120650    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-l58p5"
Sep 05 19:53:42.120788 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:42.120663    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-l58p5" event={"ID":"a7013db1-e2cf-4c0c-8116-be49d108151d","Type":"ContainerDied","Data":"eff1a25241539d89e881a9c641e139a3b0341cdc08531e9f546e6e9c6ffa6d36"}
Sep 05 19:53:42.120788 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:42.120728    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-l58p5" event={"ID":"a7013db1-e2cf-4c0c-8116-be49d108151d","Type":"ContainerDied","Data":"df59a31481247ee0c99c443381260159bcf706ea2b3a931bd806a6fb1f152652"}
Sep 05 19:53:42.120788 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:42.120759    2583 scope.go:117] "RemoveContainer" containerID="eff1a25241539d89e881a9c641e139a3b0341cdc08531e9f546e6e9c6ffa6d36"
Sep 05 19:53:42.122723 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:42.122698    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-kg8m8" event={"ID":"3259161d-2c56-4705-aca8-ff45669a1915","Type":"ContainerStarted","Data":"55756185eff23eeaabaeb5b7b6c6ffe1eb461d42e11239631d565d9b5c068f54"}
Sep 05 19:53:42.122848 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:42.122750    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-kg8m8" event={"ID":"3259161d-2c56-4705-aca8-ff45669a1915","Type":"ContainerStarted","Data":"d06aab4a8d88959e0fcf09b4dc631424ef8f3985331b0befea30c34ace168d89"}
Sep 05 19:53:42.122848 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:42.122797    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-kg8m8"
Sep 05 19:53:42.129648 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:42.129627    2583 scope.go:117] "RemoveContainer" containerID="eff1a25241539d89e881a9c641e139a3b0341cdc08531e9f546e6e9c6ffa6d36"
Sep 05 19:53:42.129924 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:42.129900    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eff1a25241539d89e881a9c641e139a3b0341cdc08531e9f546e6e9c6ffa6d36\": container with ID starting with eff1a25241539d89e881a9c641e139a3b0341cdc08531e9f546e6e9c6ffa6d36 not found: ID does not exist" containerID="eff1a25241539d89e881a9c641e139a3b0341cdc08531e9f546e6e9c6ffa6d36"
Sep 05 19:53:42.130021 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:42.129934    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff1a25241539d89e881a9c641e139a3b0341cdc08531e9f546e6e9c6ffa6d36"} err="failed to get container status \"eff1a25241539d89e881a9c641e139a3b0341cdc08531e9f546e6e9c6ffa6d36\": rpc error: code = NotFound desc = could not find container \"eff1a25241539d89e881a9c641e139a3b0341cdc08531e9f546e6e9c6ffa6d36\": container with ID starting with eff1a25241539d89e881a9c641e139a3b0341cdc08531e9f546e6e9c6ffa6d36 not found: ID does not exist"
Sep 05 19:53:42.147683 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:42.147633    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-kg8m8" podStartSLOduration=1.147618665 podStartE2EDuration="1.147618665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:53:42.144613468 +0000 UTC m=+621.174245369" watchObservedRunningTime="2026-09-05 19:53:42.147618665 +0000 UTC m=+621.177250571"
Sep 05 19:53:42.161591 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:42.161560    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-l58p5"]
Sep 05 19:53:42.165626 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:42.165601    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-l58p5"]
Sep 05 19:53:43.435282 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:43.435245    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7013db1-e2cf-4c0c-8116-be49d108151d" path="/var/lib/kubelet/pods/a7013db1-e2cf-4c0c-8116-be49d108151d/volumes"
Sep 05 19:53:53.131685 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:53.131652    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-kg8m8"
Sep 05 19:53:53.183983 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:53.183952    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-w8cvv"]
Sep 05 19:53:53.184248 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:53.184218    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-w8cvv" podUID="f91d8355-8880-4180-b0c1-112df3ee0cd6" containerName="manager" containerID="cri-o://f57495eb63c8c6eb9f0f20c2bb9668876e90be25e76e650a51b5719870cfe2c3" gracePeriod=10
Sep 05 19:53:53.434156 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:53.434130    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-w8cvv"
Sep 05 19:53:53.629689 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:53.629656    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqb9l\" (UniqueName: \"kubernetes.io/projected/f91d8355-8880-4180-b0c1-112df3ee0cd6-kube-api-access-gqb9l\") pod \"f91d8355-8880-4180-b0c1-112df3ee0cd6\" (UID: \"f91d8355-8880-4180-b0c1-112df3ee0cd6\") "
Sep 05 19:53:53.629848 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:53.629697    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f91d8355-8880-4180-b0c1-112df3ee0cd6-extensions-socket-volume\") pod \"f91d8355-8880-4180-b0c1-112df3ee0cd6\" (UID: \"f91d8355-8880-4180-b0c1-112df3ee0cd6\") "
Sep 05 19:53:53.630038 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:53.630014    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91d8355-8880-4180-b0c1-112df3ee0cd6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f91d8355-8880-4180-b0c1-112df3ee0cd6" (UID: "f91d8355-8880-4180-b0c1-112df3ee0cd6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:53:53.631736 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:53.631704    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f91d8355-8880-4180-b0c1-112df3ee0cd6-kube-api-access-gqb9l" (OuterVolumeSpecName: "kube-api-access-gqb9l") pod "f91d8355-8880-4180-b0c1-112df3ee0cd6" (UID: "f91d8355-8880-4180-b0c1-112df3ee0cd6"). InnerVolumeSpecName "kube-api-access-gqb9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:53:53.730386 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:53.730303    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqb9l\" (UniqueName: \"kubernetes.io/projected/f91d8355-8880-4180-b0c1-112df3ee0cd6-kube-api-access-gqb9l\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:53:53.730386 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:53.730332    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f91d8355-8880-4180-b0c1-112df3ee0cd6-extensions-socket-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:53:54.192664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:54.192628    2583 generic.go:356] "Generic (PLEG): container finished" podID="f91d8355-8880-4180-b0c1-112df3ee0cd6" containerID="f57495eb63c8c6eb9f0f20c2bb9668876e90be25e76e650a51b5719870cfe2c3" exitCode=0
Sep 05 19:53:54.192664 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:54.192667    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-w8cvv"
Sep 05 19:53:54.193069 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:54.192715    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-w8cvv" event={"ID":"f91d8355-8880-4180-b0c1-112df3ee0cd6","Type":"ContainerDied","Data":"f57495eb63c8c6eb9f0f20c2bb9668876e90be25e76e650a51b5719870cfe2c3"}
Sep 05 19:53:54.193069 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:54.192765    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-w8cvv" event={"ID":"f91d8355-8880-4180-b0c1-112df3ee0cd6","Type":"ContainerDied","Data":"4d1843402d560ed750c86e9c826d6e46c429bcd68b68c9142538f31708a01712"}
Sep 05 19:53:54.193069 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:54.192789    2583 scope.go:117] "RemoveContainer" containerID="f57495eb63c8c6eb9f0f20c2bb9668876e90be25e76e650a51b5719870cfe2c3"
Sep 05 19:53:54.202287 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:54.202155    2583 scope.go:117] "RemoveContainer" containerID="f57495eb63c8c6eb9f0f20c2bb9668876e90be25e76e650a51b5719870cfe2c3"
Sep 05 19:53:54.202448 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:53:54.202421    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f57495eb63c8c6eb9f0f20c2bb9668876e90be25e76e650a51b5719870cfe2c3\": container with ID starting with f57495eb63c8c6eb9f0f20c2bb9668876e90be25e76e650a51b5719870cfe2c3 not found: ID does not exist" containerID="f57495eb63c8c6eb9f0f20c2bb9668876e90be25e76e650a51b5719870cfe2c3"
Sep 05 19:53:54.202517 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:54.202455    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f57495eb63c8c6eb9f0f20c2bb9668876e90be25e76e650a51b5719870cfe2c3"} err="failed to get container status \"f57495eb63c8c6eb9f0f20c2bb9668876e90be25e76e650a51b5719870cfe2c3\": rpc error: code = NotFound desc = could not find container \"f57495eb63c8c6eb9f0f20c2bb9668876e90be25e76e650a51b5719870cfe2c3\": container with ID starting with f57495eb63c8c6eb9f0f20c2bb9668876e90be25e76e650a51b5719870cfe2c3 not found: ID does not exist"
Sep 05 19:53:54.214457 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:54.214433    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-w8cvv"]
Sep 05 19:53:54.219699 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:54.219675    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-w8cvv"]
Sep 05 19:53:55.434301 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:55.434266    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f91d8355-8880-4180-b0c1-112df3ee0cd6" path="/var/lib/kubelet/pods/f91d8355-8880-4180-b0c1-112df3ee0cd6/volumes"
Sep 05 19:53:57.457244 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.457208    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"]
Sep 05 19:53:57.457609 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.457581    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f91d8355-8880-4180-b0c1-112df3ee0cd6" containerName="manager"
Sep 05 19:53:57.457609 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.457593    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f91d8355-8880-4180-b0c1-112df3ee0cd6" containerName="manager"
Sep 05 19:53:57.457679 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.457661    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f91d8355-8880-4180-b0c1-112df3ee0cd6" containerName="manager"
Sep 05 19:53:57.629004 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.628972    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"]
Sep 05 19:53:57.629159 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.629083    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.631575 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.631553    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-xcmqj\""
Sep 05 19:53:57.666597 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.666568    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d56147e-8a67-48db-8780-0c686acf83da-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.666742 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.666601    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d56147e-8a67-48db-8780-0c686acf83da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.666742 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.666625    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56147e-8a67-48db-8780-0c686acf83da-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.666742 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.666698    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d56147e-8a67-48db-8780-0c686acf83da-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.666861 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.666742    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d56147e-8a67-48db-8780-0c686acf83da-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.666861 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.666768    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxv28\" (UniqueName: \"kubernetes.io/projected/0d56147e-8a67-48db-8780-0c686acf83da-kube-api-access-lxv28\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.666861 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.666795    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56147e-8a67-48db-8780-0c686acf83da-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.666861 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.666810    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d56147e-8a67-48db-8780-0c686acf83da-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.666861 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.666832    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d56147e-8a67-48db-8780-0c686acf83da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.767577 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.767491    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d56147e-8a67-48db-8780-0c686acf83da-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.767577 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.767527    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d56147e-8a67-48db-8780-0c686acf83da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.767577 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.767551    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56147e-8a67-48db-8780-0c686acf83da-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.767577 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.767571    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d56147e-8a67-48db-8780-0c686acf83da-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.767867 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.767605    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d56147e-8a67-48db-8780-0c686acf83da-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.767867 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.767632    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxv28\" (UniqueName: \"kubernetes.io/projected/0d56147e-8a67-48db-8780-0c686acf83da-kube-api-access-lxv28\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.767867 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.767669    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56147e-8a67-48db-8780-0c686acf83da-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.767867 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.767692    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d56147e-8a67-48db-8780-0c686acf83da-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.767867 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.767722    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d56147e-8a67-48db-8780-0c686acf83da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.768164 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.767987    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56147e-8a67-48db-8780-0c686acf83da-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.768164 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.768005    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d56147e-8a67-48db-8780-0c686acf83da-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.768278 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.768202    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d56147e-8a67-48db-8780-0c686acf83da-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.768278 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.768218    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56147e-8a67-48db-8780-0c686acf83da-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.768490 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.768465    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d56147e-8a67-48db-8780-0c686acf83da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.769918 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.769890    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d56147e-8a67-48db-8780-0c686acf83da-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.770135 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.770103    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d56147e-8a67-48db-8780-0c686acf83da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.775192 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.775167    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d56147e-8a67-48db-8780-0c686acf83da-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.775336 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.775316    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxv28\" (UniqueName: \"kubernetes.io/projected/0d56147e-8a67-48db-8780-0c686acf83da-kube-api-access-lxv28\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h7l4s\" (UID: \"0d56147e-8a67-48db-8780-0c686acf83da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:57.939361 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:57.939326    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:58.214880 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:58.214855    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"]
Sep 05 19:53:58.217018 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:53:58.216994    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d56147e_8a67_48db_8780_0c686acf83da.slice/crio-a0c7b5a7ebea3389e1114bdb2883c083cf2f9d5a22c629a3cd2ef64878dc0e51 WatchSource:0}: Error finding container a0c7b5a7ebea3389e1114bdb2883c083cf2f9d5a22c629a3cd2ef64878dc0e51: Status 404 returned error can't find the container with id a0c7b5a7ebea3389e1114bdb2883c083cf2f9d5a22c629a3cd2ef64878dc0e51
Sep 05 19:53:58.219531 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:58.219500    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:53:58.219597 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:58.219580    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:53:58.219635 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:58.219621    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:53:59.222501 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:59.222461    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s" event={"ID":"0d56147e-8a67-48db-8780-0c686acf83da","Type":"ContainerStarted","Data":"5afcf5455e3a808dd661295cf2a8126e94fe81c170b143393b1d649e5220ead7"}
Sep 05 19:53:59.222501 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:59.222510    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s" event={"ID":"0d56147e-8a67-48db-8780-0c686acf83da","Type":"ContainerStarted","Data":"a0c7b5a7ebea3389e1114bdb2883c083cf2f9d5a22c629a3cd2ef64878dc0e51"}
Sep 05 19:53:59.243968 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:59.243907    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s" podStartSLOduration=2.243893299 podStartE2EDuration="2.243893299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:53:59.243776822 +0000 UTC m=+638.273408728" watchObservedRunningTime="2026-09-05 19:53:59.243893299 +0000 UTC m=+638.273525204"
Sep 05 19:53:59.939754 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:59.939709    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:53:59.944617 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:53:59.944586    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:54:00.228309 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:00.228221    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:54:00.229345 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:00.229320    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:54:12.891539 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:12.891506    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-x9fl9"]
Sep 05 19:54:12.896024 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:12.895999    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-x9fl9"
Sep 05 19:54:12.898236 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:12.898213    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2zhs7\""
Sep 05 19:54:12.899154 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:12.899104    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-x9fl9"]
Sep 05 19:54:12.986053 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:12.986018    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfhgm\" (UniqueName: \"kubernetes.io/projected/96b5f9bc-9656-4be1-ad58-824043698f8e-kube-api-access-gfhgm\") pod \"authorino-c548b5d7d-x9fl9\" (UID: \"96b5f9bc-9656-4be1-ad58-824043698f8e\") " pod="kuadrant-system/authorino-c548b5d7d-x9fl9"
Sep 05 19:54:13.086801 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:13.086764    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfhgm\" (UniqueName: \"kubernetes.io/projected/96b5f9bc-9656-4be1-ad58-824043698f8e-kube-api-access-gfhgm\") pod \"authorino-c548b5d7d-x9fl9\" (UID: \"96b5f9bc-9656-4be1-ad58-824043698f8e\") " pod="kuadrant-system/authorino-c548b5d7d-x9fl9"
Sep 05 19:54:13.098707 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:13.098674    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfhgm\" (UniqueName: \"kubernetes.io/projected/96b5f9bc-9656-4be1-ad58-824043698f8e-kube-api-access-gfhgm\") pod \"authorino-c548b5d7d-x9fl9\" (UID: \"96b5f9bc-9656-4be1-ad58-824043698f8e\") " pod="kuadrant-system/authorino-c548b5d7d-x9fl9"
Sep 05 19:54:13.206784 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:13.206703    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-x9fl9"
Sep 05 19:54:13.477141 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:13.477098    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-x9fl9"]
Sep 05 19:54:13.478971 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:54:13.478946    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96b5f9bc_9656_4be1_ad58_824043698f8e.slice/crio-bb428eeeb18a864ae302ab67b7481668047259a8438eabc641eeb8589770e11a WatchSource:0}: Error finding container bb428eeeb18a864ae302ab67b7481668047259a8438eabc641eeb8589770e11a: Status 404 returned error can't find the container with id bb428eeeb18a864ae302ab67b7481668047259a8438eabc641eeb8589770e11a
Sep 05 19:54:14.311001 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:14.310962    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-x9fl9" event={"ID":"96b5f9bc-9656-4be1-ad58-824043698f8e","Type":"ContainerStarted","Data":"bb428eeeb18a864ae302ab67b7481668047259a8438eabc641eeb8589770e11a"}
Sep 05 19:54:16.326698 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:16.326659    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-x9fl9" event={"ID":"96b5f9bc-9656-4be1-ad58-824043698f8e","Type":"ContainerStarted","Data":"f58eed51d4575f0069bb423b7f50435daa771318e8b26d02a302299483a21c86"}
Sep 05 19:54:16.342179 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:16.342126    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-x9fl9" podStartSLOduration=2.4222313509999998 podStartE2EDuration="4.34209337s" totalImagesPullingTime="1.919862019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:54:16.341350489 +0000 UTC m=+655.370982395" watchObservedRunningTime="2026-09-05 19:54:16.34209337 +0000 UTC m=+655.371725311"
Sep 05 19:54:44.769053 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:44.769016    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-64c6b6c7cc-jt68r"]
Sep 05 19:54:44.774288 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:44.774269    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64c6b6c7cc-jt68r"
Sep 05 19:54:44.776660 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:44.776635    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 05 19:54:44.778582 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:44.778555    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64c6b6c7cc-jt68r"]
Sep 05 19:54:44.870740 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:44.870708    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/248cf7f8-a883-4c92-a3fb-7b83597c8403-tls-cert\") pod \"authorino-64c6b6c7cc-jt68r\" (UID: \"248cf7f8-a883-4c92-a3fb-7b83597c8403\") " pod="kuadrant-system/authorino-64c6b6c7cc-jt68r"
Sep 05 19:54:44.870916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:44.870840    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czl9v\" (UniqueName: \"kubernetes.io/projected/248cf7f8-a883-4c92-a3fb-7b83597c8403-kube-api-access-czl9v\") pod \"authorino-64c6b6c7cc-jt68r\" (UID: \"248cf7f8-a883-4c92-a3fb-7b83597c8403\") " pod="kuadrant-system/authorino-64c6b6c7cc-jt68r"
Sep 05 19:54:44.972220 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:44.972178    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czl9v\" (UniqueName: \"kubernetes.io/projected/248cf7f8-a883-4c92-a3fb-7b83597c8403-kube-api-access-czl9v\") pod \"authorino-64c6b6c7cc-jt68r\" (UID: \"248cf7f8-a883-4c92-a3fb-7b83597c8403\") " pod="kuadrant-system/authorino-64c6b6c7cc-jt68r"
Sep 05 19:54:44.972433 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:44.972242    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/248cf7f8-a883-4c92-a3fb-7b83597c8403-tls-cert\") pod \"authorino-64c6b6c7cc-jt68r\" (UID: \"248cf7f8-a883-4c92-a3fb-7b83597c8403\") " pod="kuadrant-system/authorino-64c6b6c7cc-jt68r"
Sep 05 19:54:44.974869 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:44.974843    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/248cf7f8-a883-4c92-a3fb-7b83597c8403-tls-cert\") pod \"authorino-64c6b6c7cc-jt68r\" (UID: \"248cf7f8-a883-4c92-a3fb-7b83597c8403\") " pod="kuadrant-system/authorino-64c6b6c7cc-jt68r"
Sep 05 19:54:44.980400 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:44.980368    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czl9v\" (UniqueName: \"kubernetes.io/projected/248cf7f8-a883-4c92-a3fb-7b83597c8403-kube-api-access-czl9v\") pod \"authorino-64c6b6c7cc-jt68r\" (UID: \"248cf7f8-a883-4c92-a3fb-7b83597c8403\") " pod="kuadrant-system/authorino-64c6b6c7cc-jt68r"
Sep 05 19:54:45.085078 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:45.085049    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64c6b6c7cc-jt68r"
Sep 05 19:54:45.353534 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:45.353509    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64c6b6c7cc-jt68r"]
Sep 05 19:54:45.356093 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:54:45.356064    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod248cf7f8_a883_4c92_a3fb_7b83597c8403.slice/crio-7f52a70deaab88ce5d873077ceaed2d9009956c8008f2635392ee900deac5d3e WatchSource:0}: Error finding container 7f52a70deaab88ce5d873077ceaed2d9009956c8008f2635392ee900deac5d3e: Status 404 returned error can't find the container with id 7f52a70deaab88ce5d873077ceaed2d9009956c8008f2635392ee900deac5d3e
Sep 05 19:54:45.357461 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:45.357441    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 19:54:45.482716 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:45.482681    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64c6b6c7cc-jt68r" event={"ID":"248cf7f8-a883-4c92-a3fb-7b83597c8403","Type":"ContainerStarted","Data":"7f52a70deaab88ce5d873077ceaed2d9009956c8008f2635392ee900deac5d3e"}
Sep 05 19:54:46.489596 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:46.489564    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64c6b6c7cc-jt68r" event={"ID":"248cf7f8-a883-4c92-a3fb-7b83597c8403","Type":"ContainerStarted","Data":"13f6846328ab2e6b4208c8e023022ac4bcaeb41d93eff35364e826f96274bcc3"}
Sep 05 19:54:46.507855 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:46.507804    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-64c6b6c7cc-jt68r" podStartSLOduration=2.182234206 podStartE2EDuration="2.507787856s" totalImagesPullingTime="325.55365ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:54:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:54:46.505876838 +0000 UTC m=+685.535508744" watchObservedRunningTime="2026-09-05 19:54:46.507787856 +0000 UTC m=+685.537419760"
Sep 05 19:54:46.535360 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:46.535328    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-x9fl9"]
Sep 05 19:54:46.535582 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:46.535518    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-x9fl9" podUID="96b5f9bc-9656-4be1-ad58-824043698f8e" containerName="authorino" containerID="cri-o://f58eed51d4575f0069bb423b7f50435daa771318e8b26d02a302299483a21c86" gracePeriod=30
Sep 05 19:54:46.793172 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:46.793146    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-x9fl9"
Sep 05 19:54:46.890095 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:46.890064    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfhgm\" (UniqueName: \"kubernetes.io/projected/96b5f9bc-9656-4be1-ad58-824043698f8e-kube-api-access-gfhgm\") pod \"96b5f9bc-9656-4be1-ad58-824043698f8e\" (UID: \"96b5f9bc-9656-4be1-ad58-824043698f8e\") "
Sep 05 19:54:46.892148 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:46.892102    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b5f9bc-9656-4be1-ad58-824043698f8e-kube-api-access-gfhgm" (OuterVolumeSpecName: "kube-api-access-gfhgm") pod "96b5f9bc-9656-4be1-ad58-824043698f8e" (UID: "96b5f9bc-9656-4be1-ad58-824043698f8e"). InnerVolumeSpecName "kube-api-access-gfhgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:54:46.991132 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:46.991063    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfhgm\" (UniqueName: \"kubernetes.io/projected/96b5f9bc-9656-4be1-ad58-824043698f8e-kube-api-access-gfhgm\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:54:47.496397 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:47.496356    2583 generic.go:356] "Generic (PLEG): container finished" podID="96b5f9bc-9656-4be1-ad58-824043698f8e" containerID="f58eed51d4575f0069bb423b7f50435daa771318e8b26d02a302299483a21c86" exitCode=0
Sep 05 19:54:47.496815 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:47.496403    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-x9fl9"
Sep 05 19:54:47.496815 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:47.496435    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-x9fl9" event={"ID":"96b5f9bc-9656-4be1-ad58-824043698f8e","Type":"ContainerDied","Data":"f58eed51d4575f0069bb423b7f50435daa771318e8b26d02a302299483a21c86"}
Sep 05 19:54:47.496815 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:47.496482    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-x9fl9" event={"ID":"96b5f9bc-9656-4be1-ad58-824043698f8e","Type":"ContainerDied","Data":"bb428eeeb18a864ae302ab67b7481668047259a8438eabc641eeb8589770e11a"}
Sep 05 19:54:47.496815 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:47.496500    2583 scope.go:117] "RemoveContainer" containerID="f58eed51d4575f0069bb423b7f50435daa771318e8b26d02a302299483a21c86"
Sep 05 19:54:47.505025 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:47.505004    2583 scope.go:117] "RemoveContainer" containerID="f58eed51d4575f0069bb423b7f50435daa771318e8b26d02a302299483a21c86"
Sep 05 19:54:47.505316 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:54:47.505295    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f58eed51d4575f0069bb423b7f50435daa771318e8b26d02a302299483a21c86\": container with ID starting with f58eed51d4575f0069bb423b7f50435daa771318e8b26d02a302299483a21c86 not found: ID does not exist" containerID="f58eed51d4575f0069bb423b7f50435daa771318e8b26d02a302299483a21c86"
Sep 05 19:54:47.505368 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:47.505325    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f58eed51d4575f0069bb423b7f50435daa771318e8b26d02a302299483a21c86"} err="failed to get container status \"f58eed51d4575f0069bb423b7f50435daa771318e8b26d02a302299483a21c86\": rpc error: code = NotFound desc = could not find container \"f58eed51d4575f0069bb423b7f50435daa771318e8b26d02a302299483a21c86\": container with ID starting with f58eed51d4575f0069bb423b7f50435daa771318e8b26d02a302299483a21c86 not found: ID does not exist"
Sep 05 19:54:47.513036 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:47.513013    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-x9fl9"]
Sep 05 19:54:47.519684 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:47.519660    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-x9fl9"]
Sep 05 19:54:49.435244 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:54:49.435211    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96b5f9bc-9656-4be1-ad58-824043698f8e" path="/var/lib/kubelet/pods/96b5f9bc-9656-4be1-ad58-824043698f8e/volumes"
Sep 05 19:55:29.943826 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:29.943791    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-75cn5"]
Sep 05 19:55:29.944227 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:29.944165    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96b5f9bc-9656-4be1-ad58-824043698f8e" containerName="authorino"
Sep 05 19:55:29.944227 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:29.944176    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="96b5f9bc-9656-4be1-ad58-824043698f8e" containerName="authorino"
Sep 05 19:55:29.944303 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:29.944236    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="96b5f9bc-9656-4be1-ad58-824043698f8e" containerName="authorino"
Sep 05 19:55:29.948318 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:29.948296    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-75cn5"
Sep 05 19:55:29.950570 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:29.950544    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-znz9n\""
Sep 05 19:55:29.950703 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:29.950613    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 05 19:55:29.960601 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:29.960575    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-75cn5"]
Sep 05 19:55:30.070339 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:30.070298    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb5f281-7a4f-4568-adf6-945df3d1e160-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-75cn5\" (UID: \"bbb5f281-7a4f-4568-adf6-945df3d1e160\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-75cn5"
Sep 05 19:55:30.070541 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:30.070366    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znpn4\" (UniqueName: \"kubernetes.io/projected/bbb5f281-7a4f-4568-adf6-945df3d1e160-kube-api-access-znpn4\") pod \"payload-pre-processing-59fbc47bf8-75cn5\" (UID: \"bbb5f281-7a4f-4568-adf6-945df3d1e160\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-75cn5"
Sep 05 19:55:30.171609 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:30.171565    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb5f281-7a4f-4568-adf6-945df3d1e160-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-75cn5\" (UID: \"bbb5f281-7a4f-4568-adf6-945df3d1e160\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-75cn5"
Sep 05 19:55:30.171781 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:30.171643    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znpn4\" (UniqueName: \"kubernetes.io/projected/bbb5f281-7a4f-4568-adf6-945df3d1e160-kube-api-access-znpn4\") pod \"payload-pre-processing-59fbc47bf8-75cn5\" (UID: \"bbb5f281-7a4f-4568-adf6-945df3d1e160\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-75cn5"
Sep 05 19:55:30.172375 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:30.172354    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbb5f281-7a4f-4568-adf6-945df3d1e160-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-75cn5\" (UID: \"bbb5f281-7a4f-4568-adf6-945df3d1e160\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-75cn5"
Sep 05 19:55:30.183169 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:30.183134    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znpn4\" (UniqueName: \"kubernetes.io/projected/bbb5f281-7a4f-4568-adf6-945df3d1e160-kube-api-access-znpn4\") pod \"payload-pre-processing-59fbc47bf8-75cn5\" (UID: \"bbb5f281-7a4f-4568-adf6-945df3d1e160\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-75cn5"
Sep 05 19:55:30.261177 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:30.261073    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-75cn5"
Sep 05 19:55:30.551232 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:30.551205    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-75cn5"]
Sep 05 19:55:30.552919 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:55:30.552890    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb5f281_7a4f_4568_adf6_945df3d1e160.slice/crio-a48106d923df2ca8220b289721b5b4c4605b64da8d9bb3c45d8c7f4fee638963 WatchSource:0}: Error finding container a48106d923df2ca8220b289721b5b4c4605b64da8d9bb3c45d8c7f4fee638963: Status 404 returned error can't find the container with id a48106d923df2ca8220b289721b5b4c4605b64da8d9bb3c45d8c7f4fee638963
Sep 05 19:55:30.729458 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:30.729427    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-75cn5" event={"ID":"bbb5f281-7a4f-4568-adf6-945df3d1e160","Type":"ContainerStarted","Data":"a48106d923df2ca8220b289721b5b4c4605b64da8d9bb3c45d8c7f4fee638963"}
Sep 05 19:55:32.742040 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:32.742003    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-75cn5" event={"ID":"bbb5f281-7a4f-4568-adf6-945df3d1e160","Type":"ContainerStarted","Data":"5259b3b97f69bfeef224d3a6b02df7e9f17c24dbaf20df5781880c92b92f93dc"}
Sep 05 19:55:32.742430 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:32.742147    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-75cn5"
Sep 05 19:55:32.768239 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:32.768168    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-75cn5" podStartSLOduration=2.017274104 podStartE2EDuration="3.768149969s" totalImagesPullingTime="1.750875865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:55:32.764738514 +0000 UTC m=+731.794370420" watchObservedRunningTime="2026-09-05 19:55:32.768149969 +0000 UTC m=+731.797781875"
Sep 05 19:55:39.784056 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:39.784014    2583 generic.go:356] "Generic (PLEG): container finished" podID="0d56147e-8a67-48db-8780-0c686acf83da" containerID="5afcf5455e3a808dd661295cf2a8126e94fe81c170b143393b1d649e5220ead7" exitCode=0
Sep 05 19:55:39.784478 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:39.784074    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s" event={"ID":"0d56147e-8a67-48db-8780-0c686acf83da","Type":"ContainerDied","Data":"5afcf5455e3a808dd661295cf2a8126e94fe81c170b143393b1d649e5220ead7"}
Sep 05 19:55:39.784526 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:39.784476    2583 scope.go:117] "RemoveContainer" containerID="5afcf5455e3a808dd661295cf2a8126e94fe81c170b143393b1d649e5220ead7"
Sep 05 19:55:39.785561 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:39.785529    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:55:39.785631 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:39.785615    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:55:39.785681 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:39.785666    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:55:40.794861 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:40.794821    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s" event={"ID":"0d56147e-8a67-48db-8780-0c686acf83da","Type":"ContainerStarted","Data":"aa90d4639458f009a813e810f240a2c3a4b0ba2265145c819d30be8ffc6ae8b5"}
Sep 05 19:55:40.940246 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:40.940209    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:55:40.941220 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:40.941190    2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-h7l4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 05 19:55:40.941285 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:40.941250    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s" podUID="0d56147e-8a67-48db-8780-0c686acf83da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 05 19:55:41.940299 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:41.940264    2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-h7l4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 05 19:55:41.940695 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:41.940318    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s" podUID="0d56147e-8a67-48db-8780-0c686acf83da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 05 19:55:42.940458 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:42.940420    2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-h7l4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 05 19:55:42.940847 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:42.940476    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s" podUID="0d56147e-8a67-48db-8780-0c686acf83da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 05 19:55:43.749497 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:43.749465    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-75cn5"
Sep 05 19:55:43.940531 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:43.940496    2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-h7l4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 05 19:55:43.940929 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:43.940551    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s" podUID="0d56147e-8a67-48db-8780-0c686acf83da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 05 19:55:44.940276 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:44.940239    2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-h7l4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 05 19:55:44.940456 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:44.940294    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s" podUID="0d56147e-8a67-48db-8780-0c686acf83da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 05 19:55:45.229196 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:45.229086    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:55:45.940359 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:45.940315    2583 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-h7l4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 05 19:55:45.940540 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:45.940397    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s" podUID="0d56147e-8a67-48db-8780-0c686acf83da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 05 19:55:46.943346 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:46.943314    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:55:46.944192 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:55:46.944172    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h7l4s"
Sep 05 19:56:01.127918 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.127881    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"]
Sep 05 19:56:01.131609 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.131588    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"
Sep 05 19:56:01.134768 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.134745    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 05 19:56:01.134937 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.134924    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dnwjp\""
Sep 05 19:56:01.135012 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.134958    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 05 19:56:01.135012 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.134975    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 05 19:56:01.145485 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.145454    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"]
Sep 05 19:56:01.246245 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.246206    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh\" (UID: \"ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"
Sep 05 19:56:01.246438 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.246258    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh\" (UID: \"ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"
Sep 05 19:56:01.246438 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.246325    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh\" (UID: \"ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"
Sep 05 19:56:01.246438 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.246409    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh\" (UID: \"ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"
Sep 05 19:56:01.246588 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.246453    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blgn5\" (UniqueName: \"kubernetes.io/projected/ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c-kube-api-access-blgn5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh\" (UID: \"ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"
Sep 05 19:56:01.246588 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.246502    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh\" (UID: \"ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"
Sep 05 19:56:01.347455 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.347418    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh\" (UID: \"ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"
Sep 05 19:56:01.347455 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.347462    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blgn5\" (UniqueName: \"kubernetes.io/projected/ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c-kube-api-access-blgn5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh\" (UID: \"ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"
Sep 05 19:56:01.347700 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.347590    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh\" (UID: \"ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"
Sep 05 19:56:01.347700 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.347681    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh\" (UID: \"ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"
Sep 05 19:56:01.347807 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.347739    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh\" (UID: \"ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"
Sep 05 19:56:01.347807 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.347770    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh\" (UID: \"ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"
Sep 05 19:56:01.347943 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.347925    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh\" (UID: \"ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"
Sep 05 19:56:01.348004 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.347988    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh\" (UID: \"ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"
Sep 05 19:56:01.348075 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.348057    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh\" (UID: \"ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"
Sep 05 19:56:01.349985 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.349964    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh\" (UID: \"ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"
Sep 05 19:56:01.350093 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.349967    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh\" (UID: \"ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"
Sep 05 19:56:01.356760 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.356733    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blgn5\" (UniqueName: \"kubernetes.io/projected/ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c-kube-api-access-blgn5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh\" (UID: \"ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"
Sep 05 19:56:01.441914 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.441828    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"
Sep 05 19:56:01.727278 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.727250    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"]
Sep 05 19:56:01.729053 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:56:01.729026    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2eb6d6_81b7_4c8a_8dcf_6e40c3fd1a5c.slice/crio-e1a64ca0ae93422668d39f417f1d2e0590f36f7021a07dc046d7f0894af8f4b9 WatchSource:0}: Error finding container e1a64ca0ae93422668d39f417f1d2e0590f36f7021a07dc046d7f0894af8f4b9: Status 404 returned error can't find the container with id e1a64ca0ae93422668d39f417f1d2e0590f36f7021a07dc046d7f0894af8f4b9
Sep 05 19:56:01.917998 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:01.917955    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh" event={"ID":"ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c","Type":"ContainerStarted","Data":"e1a64ca0ae93422668d39f417f1d2e0590f36f7021a07dc046d7f0894af8f4b9"}
Sep 05 19:56:03.933970 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:03.933925    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh" event={"ID":"ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c","Type":"ContainerStarted","Data":"cebe8a7b32e13b34096579cdc718c9d1718829c91d636c074e79cf24a27aa6de"}
Sep 05 19:56:03.954273 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:03.954214    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh" podStartSLOduration=1.723717569 podStartE2EDuration="2.954196738s" totalImagesPullingTime="1.230479169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:56:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:56:03.953261596 +0000 UTC m=+762.982893501" watchObservedRunningTime="2026-09-05 19:56:03.954196738 +0000 UTC m=+762.983828688"
Sep 05 19:56:11.442551 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:11.442515    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"
Sep 05 19:56:11.442943 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:11.442573    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"
Sep 05 19:56:11.455488 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:11.455463    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"
Sep 05 19:56:11.990557 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:56:11.990526    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh"
Sep 05 19:58:21.419175 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:21.419080    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf9tj_f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b/ovn-acl-logging/0.log"
Sep 05 19:58:21.422276 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:21.422255    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf9tj_f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b/ovn-acl-logging/0.log"
Sep 05 19:58:38.431302 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:38.431266    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f67c45996-4jg96"]
Sep 05 19:58:38.434725 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:38.434708    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f67c45996-4jg96"
Sep 05 19:58:38.442319 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:38.441972    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f67c45996-4jg96"]
Sep 05 19:58:38.529920 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:38.529886    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/720e095e-4556-4efc-8b0c-b1b68b037afa-tls-cert\") pod \"authorino-5f67c45996-4jg96\" (UID: \"720e095e-4556-4efc-8b0c-b1b68b037afa\") " pod="kuadrant-system/authorino-5f67c45996-4jg96"
Sep 05 19:58:38.530068 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:38.529931    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz6js\" (UniqueName: \"kubernetes.io/projected/720e095e-4556-4efc-8b0c-b1b68b037afa-kube-api-access-sz6js\") pod \"authorino-5f67c45996-4jg96\" (UID: \"720e095e-4556-4efc-8b0c-b1b68b037afa\") " pod="kuadrant-system/authorino-5f67c45996-4jg96"
Sep 05 19:58:38.630846 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:38.630815    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz6js\" (UniqueName: \"kubernetes.io/projected/720e095e-4556-4efc-8b0c-b1b68b037afa-kube-api-access-sz6js\") pod \"authorino-5f67c45996-4jg96\" (UID: \"720e095e-4556-4efc-8b0c-b1b68b037afa\") " pod="kuadrant-system/authorino-5f67c45996-4jg96"
Sep 05 19:58:38.631000 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:38.630976    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/720e095e-4556-4efc-8b0c-b1b68b037afa-tls-cert\") pod \"authorino-5f67c45996-4jg96\" (UID: \"720e095e-4556-4efc-8b0c-b1b68b037afa\") " pod="kuadrant-system/authorino-5f67c45996-4jg96"
Sep 05 19:58:38.633396 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:38.633374    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/720e095e-4556-4efc-8b0c-b1b68b037afa-tls-cert\") pod \"authorino-5f67c45996-4jg96\" (UID: \"720e095e-4556-4efc-8b0c-b1b68b037afa\") " pod="kuadrant-system/authorino-5f67c45996-4jg96"
Sep 05 19:58:38.639304 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:38.639279    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz6js\" (UniqueName: \"kubernetes.io/projected/720e095e-4556-4efc-8b0c-b1b68b037afa-kube-api-access-sz6js\") pod \"authorino-5f67c45996-4jg96\" (UID: \"720e095e-4556-4efc-8b0c-b1b68b037afa\") " pod="kuadrant-system/authorino-5f67c45996-4jg96"
Sep 05 19:58:38.745049 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:38.744968    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f67c45996-4jg96"
Sep 05 19:58:39.017125 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:39.017081    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f67c45996-4jg96"]
Sep 05 19:58:39.019054 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:58:39.019026    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod720e095e_4556_4efc_8b0c_b1b68b037afa.slice/crio-e6401dab31d4444a00f19e523940d795cfd6fc60d5cb1b1414696dceede64659 WatchSource:0}: Error finding container e6401dab31d4444a00f19e523940d795cfd6fc60d5cb1b1414696dceede64659: Status 404 returned error can't find the container with id e6401dab31d4444a00f19e523940d795cfd6fc60d5cb1b1414696dceede64659
Sep 05 19:58:39.792869 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:39.792831    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f67c45996-4jg96" event={"ID":"720e095e-4556-4efc-8b0c-b1b68b037afa","Type":"ContainerStarted","Data":"7778cfe4dcce5b7498100b056349e7f4b2ee388aa2adbaa3c277f3d8acbf8b77"}
Sep 05 19:58:39.792869 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:39.792879    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f67c45996-4jg96" event={"ID":"720e095e-4556-4efc-8b0c-b1b68b037afa","Type":"ContainerStarted","Data":"e6401dab31d4444a00f19e523940d795cfd6fc60d5cb1b1414696dceede64659"}
Sep 05 19:58:39.812393 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:39.812331    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5f67c45996-4jg96" podStartSLOduration=1.236395797 podStartE2EDuration="1.812314646s" totalImagesPullingTime="575.918849ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:58:39.809526937 +0000 UTC m=+918.839158844" watchObservedRunningTime="2026-09-05 19:58:39.812314646 +0000 UTC m=+918.841946550"
Sep 05 19:58:39.846810 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:39.846094    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-64c6b6c7cc-jt68r"]
Sep 05 19:58:39.847289 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:39.847253    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-64c6b6c7cc-jt68r" podUID="248cf7f8-a883-4c92-a3fb-7b83597c8403" containerName="authorino" containerID="cri-o://13f6846328ab2e6b4208c8e023022ac4bcaeb41d93eff35364e826f96274bcc3" gracePeriod=30
Sep 05 19:58:40.123031 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:40.123000    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64c6b6c7cc-jt68r"
Sep 05 19:58:40.245453 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:40.245418    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/248cf7f8-a883-4c92-a3fb-7b83597c8403-tls-cert\") pod \"248cf7f8-a883-4c92-a3fb-7b83597c8403\" (UID: \"248cf7f8-a883-4c92-a3fb-7b83597c8403\") "
Sep 05 19:58:40.245648 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:40.245525    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czl9v\" (UniqueName: \"kubernetes.io/projected/248cf7f8-a883-4c92-a3fb-7b83597c8403-kube-api-access-czl9v\") pod \"248cf7f8-a883-4c92-a3fb-7b83597c8403\" (UID: \"248cf7f8-a883-4c92-a3fb-7b83597c8403\") "
Sep 05 19:58:40.247662 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:40.247627    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248cf7f8-a883-4c92-a3fb-7b83597c8403-kube-api-access-czl9v" (OuterVolumeSpecName: "kube-api-access-czl9v") pod "248cf7f8-a883-4c92-a3fb-7b83597c8403" (UID: "248cf7f8-a883-4c92-a3fb-7b83597c8403"). InnerVolumeSpecName "kube-api-access-czl9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:58:40.256950 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:40.256903    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/248cf7f8-a883-4c92-a3fb-7b83597c8403-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "248cf7f8-a883-4c92-a3fb-7b83597c8403" (UID: "248cf7f8-a883-4c92-a3fb-7b83597c8403"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 19:58:40.346872 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:40.346827    2583 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/248cf7f8-a883-4c92-a3fb-7b83597c8403-tls-cert\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:58:40.346872 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:40.346869    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czl9v\" (UniqueName: \"kubernetes.io/projected/248cf7f8-a883-4c92-a3fb-7b83597c8403-kube-api-access-czl9v\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:58:40.800081 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:40.800041    2583 generic.go:356] "Generic (PLEG): container finished" podID="248cf7f8-a883-4c92-a3fb-7b83597c8403" containerID="13f6846328ab2e6b4208c8e023022ac4bcaeb41d93eff35364e826f96274bcc3" exitCode=0
Sep 05 19:58:40.800492 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:40.800088    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64c6b6c7cc-jt68r"
Sep 05 19:58:40.800492 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:40.800138    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64c6b6c7cc-jt68r" event={"ID":"248cf7f8-a883-4c92-a3fb-7b83597c8403","Type":"ContainerDied","Data":"13f6846328ab2e6b4208c8e023022ac4bcaeb41d93eff35364e826f96274bcc3"}
Sep 05 19:58:40.800492 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:40.800197    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64c6b6c7cc-jt68r" event={"ID":"248cf7f8-a883-4c92-a3fb-7b83597c8403","Type":"ContainerDied","Data":"7f52a70deaab88ce5d873077ceaed2d9009956c8008f2635392ee900deac5d3e"}
Sep 05 19:58:40.800492 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:40.800226    2583 scope.go:117] "RemoveContainer" containerID="13f6846328ab2e6b4208c8e023022ac4bcaeb41d93eff35364e826f96274bcc3"
Sep 05 19:58:40.811892 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:40.811618    2583 scope.go:117] "RemoveContainer" containerID="13f6846328ab2e6b4208c8e023022ac4bcaeb41d93eff35364e826f96274bcc3"
Sep 05 19:58:40.812006 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:58:40.811969    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13f6846328ab2e6b4208c8e023022ac4bcaeb41d93eff35364e826f96274bcc3\": container with ID starting with 13f6846328ab2e6b4208c8e023022ac4bcaeb41d93eff35364e826f96274bcc3 not found: ID does not exist" containerID="13f6846328ab2e6b4208c8e023022ac4bcaeb41d93eff35364e826f96274bcc3"
Sep 05 19:58:40.812006 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:40.811995    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f6846328ab2e6b4208c8e023022ac4bcaeb41d93eff35364e826f96274bcc3"} err="failed to get container status \"13f6846328ab2e6b4208c8e023022ac4bcaeb41d93eff35364e826f96274bcc3\": rpc error: code = NotFound desc = could not find container \"13f6846328ab2e6b4208c8e023022ac4bcaeb41d93eff35364e826f96274bcc3\": container with ID starting with 13f6846328ab2e6b4208c8e023022ac4bcaeb41d93eff35364e826f96274bcc3 not found: ID does not exist"
Sep 05 19:58:40.822230 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:40.822203    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-64c6b6c7cc-jt68r"]
Sep 05 19:58:40.829167 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:40.829140    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-64c6b6c7cc-jt68r"]
Sep 05 19:58:41.436089 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:41.436043    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="248cf7f8-a883-4c92-a3fb-7b83597c8403" path="/var/lib/kubelet/pods/248cf7f8-a883-4c92-a3fb-7b83597c8403/volumes"
Sep 05 19:58:56.687670 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.687633    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"]
Sep 05 19:58:56.688120 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.688086    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="248cf7f8-a883-4c92-a3fb-7b83597c8403" containerName="authorino"
Sep 05 19:58:56.688120 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.688101    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="248cf7f8-a883-4c92-a3fb-7b83597c8403" containerName="authorino"
Sep 05 19:58:56.688247 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.688208    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="248cf7f8-a883-4c92-a3fb-7b83597c8403" containerName="authorino"
Sep 05 19:58:56.691745 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.691721    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.708939 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.708911    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-588081-openshift-default-dockercfg-v7zxx\""
Sep 05 19:58:56.709666 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.709640    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"]
Sep 05 19:58:56.787684 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.787643    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-credential-socket\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.787878 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.787723    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-workload-certs\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.787957 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.787877    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56770103-b7ef-4c4c-8868-03e81ec44ad5-istio-podinfo\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.787957 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.787902    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56770103-b7ef-4c4c-8868-03e81ec44ad5-istio-token\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.787957 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.787939    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56770103-b7ef-4c4c-8868-03e81ec44ad5-istiod-ca-cert\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.788308 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.788040    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzczt\" (UniqueName: \"kubernetes.io/projected/56770103-b7ef-4c4c-8868-03e81ec44ad5-kube-api-access-vzczt\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.788308 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.788149    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-workload-socket\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.788308 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.788175    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-istio-data\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.788308 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.788210    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-istio-envoy\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.888846 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.888815    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-credential-socket\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.888846 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.888847    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-workload-certs\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.889105 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.888874    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56770103-b7ef-4c4c-8868-03e81ec44ad5-istio-podinfo\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.889105 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.888970    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56770103-b7ef-4c4c-8868-03e81ec44ad5-istio-token\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.889105 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.889009    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56770103-b7ef-4c4c-8868-03e81ec44ad5-istiod-ca-cert\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.889105 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.889045    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzczt\" (UniqueName: \"kubernetes.io/projected/56770103-b7ef-4c4c-8868-03e81ec44ad5-kube-api-access-vzczt\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.889105 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.889096    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-workload-socket\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.889412 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.889132    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-istio-data\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.889412 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.889170    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-istio-envoy\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.889569 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.889528    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-credential-socket\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.889758 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.889572    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-istio-data\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.889758 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.889602    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-workload-socket\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.889758 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.889611    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-workload-certs\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.889758 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.889756    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56770103-b7ef-4c4c-8868-03e81ec44ad5-istiod-ca-cert\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.891545 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.891519    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-istio-envoy\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.891776 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.891753    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56770103-b7ef-4c4c-8868-03e81ec44ad5-istio-podinfo\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.897883 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.897859    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56770103-b7ef-4c4c-8868-03e81ec44ad5-istio-token\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:56.897988 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:56.897972    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzczt\" (UniqueName: \"kubernetes.io/projected/56770103-b7ef-4c4c-8868-03e81ec44ad5-kube-api-access-vzczt\") pod \"e2e-inf-a-588081-openshift-default-648bd9598-kmp66\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") " pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:57.002460 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:57.002373    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:57.280263 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:57.280170    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"]
Sep 05 19:58:57.282560 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:58:57.282527    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56770103_b7ef_4c4c_8868_03e81ec44ad5.slice/crio-2107eb92739f59692854e40d141b991267098c68033376bd2e9e925eb037f6c8 WatchSource:0}: Error finding container 2107eb92739f59692854e40d141b991267098c68033376bd2e9e925eb037f6c8: Status 404 returned error can't find the container with id 2107eb92739f59692854e40d141b991267098c68033376bd2e9e925eb037f6c8
Sep 05 19:58:57.284644 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:57.284614    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:58:57.284746 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:57.284678    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:58:57.284746 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:57.284708    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:58:57.896064 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:57.896025    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66" event={"ID":"56770103-b7ef-4c4c-8868-03e81ec44ad5","Type":"ContainerStarted","Data":"00464ff44930788a5c94267fc00754c3a4d59947c73d329267cd032ecdba962d"}
Sep 05 19:58:57.896453 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:57.896089    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66" event={"ID":"56770103-b7ef-4c4c-8868-03e81ec44ad5","Type":"ContainerStarted","Data":"2107eb92739f59692854e40d141b991267098c68033376bd2e9e925eb037f6c8"}
Sep 05 19:58:57.916981 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:57.916927    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66" podStartSLOduration=1.916885175 podStartE2EDuration="1.916885175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:58:57.916697694 +0000 UTC m=+936.946329600" watchObservedRunningTime="2026-09-05 19:58:57.916885175 +0000 UTC m=+936.946517084"
Sep 05 19:58:58.003271 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:58.003238    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:58.008429 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:58.008400    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:58.127503 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:58.127469    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7854-140483173004464"]
Sep 05 19:58:58.131004 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:58.130985    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7854-140483173004464"
Sep 05 19:58:58.133174 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:58.133158    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cqkks\""
Sep 05 19:58:58.139624 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:58.139601    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7854-140483173004464"]
Sep 05 19:58:58.301086 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:58.300984    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4mmb\" (UniqueName: \"kubernetes.io/projected/969f4b2e-1a50-4271-948d-0ab2615538b1-kube-api-access-f4mmb\") pod \"test-curl-7854-140483173004464\" (UID: \"969f4b2e-1a50-4271-948d-0ab2615538b1\") " pod="opendatahub/test-curl-7854-140483173004464"
Sep 05 19:58:58.401623 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:58.401583    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4mmb\" (UniqueName: \"kubernetes.io/projected/969f4b2e-1a50-4271-948d-0ab2615538b1-kube-api-access-f4mmb\") pod \"test-curl-7854-140483173004464\" (UID: \"969f4b2e-1a50-4271-948d-0ab2615538b1\") " pod="opendatahub/test-curl-7854-140483173004464"
Sep 05 19:58:58.410043 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:58.410015    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4mmb\" (UniqueName: \"kubernetes.io/projected/969f4b2e-1a50-4271-948d-0ab2615538b1-kube-api-access-f4mmb\") pod \"test-curl-7854-140483173004464\" (UID: \"969f4b2e-1a50-4271-948d-0ab2615538b1\") " pod="opendatahub/test-curl-7854-140483173004464"
Sep 05 19:58:58.442089 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:58.442053    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7854-140483173004464"
Sep 05 19:58:58.717592 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:58.717564    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7854-140483173004464"]
Sep 05 19:58:58.719404 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:58:58.719376    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod969f4b2e_1a50_4271_948d_0ab2615538b1.slice/crio-c45a6459751eea3ed5499591f62b41a9e47aeaa8146e81593e56cb14df858c16 WatchSource:0}: Error finding container c45a6459751eea3ed5499591f62b41a9e47aeaa8146e81593e56cb14df858c16: Status 404 returned error can't find the container with id c45a6459751eea3ed5499591f62b41a9e47aeaa8146e81593e56cb14df858c16
Sep 05 19:58:58.903709 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:58.903676    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7854-140483173004464" event={"ID":"969f4b2e-1a50-4271-948d-0ab2615538b1","Type":"ContainerStarted","Data":"c45a6459751eea3ed5499591f62b41a9e47aeaa8146e81593e56cb14df858c16"}
Sep 05 19:58:58.904144 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:58.903979    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:58:58.904941 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:58:58.904921    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:59:00.920099 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:00.918909    2583 generic.go:356] "Generic (PLEG): container finished" podID="969f4b2e-1a50-4271-948d-0ab2615538b1" containerID="ea93d86a19bd2d313b24c902a4a0091923beef34a8232a8e1e6f325017378410" exitCode=0
Sep 05 19:59:00.920099 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:00.920048    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7854-140483173004464" event={"ID":"969f4b2e-1a50-4271-948d-0ab2615538b1","Type":"ContainerDied","Data":"ea93d86a19bd2d313b24c902a4a0091923beef34a8232a8e1e6f325017378410"}
Sep 05 19:59:01.712916 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:01.712872    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs"]
Sep 05 19:59:01.716316 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:01.716299    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs"
Sep 05 19:59:01.718700 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:01.718669    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-588081\""
Sep 05 19:59:01.718812 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:01.718715    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-588081-dockercfg-z8664\""
Sep 05 19:59:01.729055 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:01.729026    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs"]
Sep 05 19:59:01.747582 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:01.747545    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm"]
Sep 05 19:59:01.751052 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:01.751031    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm"
Sep 05 19:59:01.770393 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:01.770354    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm"]
Sep 05 19:59:01.833211 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:01.833176    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00416952-66bf-46d7-84bb-617343832ad5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs\" (UID: \"00416952-66bf-46d7-84bb-617343832ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs"
Sep 05 19:59:01.833407 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:01.833236    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6pt6\" (UniqueName: \"kubernetes.io/projected/00416952-66bf-46d7-84bb-617343832ad5-kube-api-access-l6pt6\") pod \"payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs\" (UID: \"00416952-66bf-46d7-84bb-617343832ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs"
Sep 05 19:59:01.833407 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:01.833349    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d96c4ef-3daa-42ae-af20-d4640421f102-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-588081-59665c9769-gn8pm\" (UID: \"3d96c4ef-3daa-42ae-af20-d4640421f102\") " pod="openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm"
Sep 05 19:59:01.833512 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:01.833407    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h956b\" (UniqueName: \"kubernetes.io/projected/3d96c4ef-3daa-42ae-af20-d4640421f102-kube-api-access-h956b\") pod \"payload-processing-e2e-inf-a-588081-59665c9769-gn8pm\" (UID: \"3d96c4ef-3daa-42ae-af20-d4640421f102\") " pod="openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm"
Sep 05 19:59:01.934626 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:01.934593    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00416952-66bf-46d7-84bb-617343832ad5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs\" (UID: \"00416952-66bf-46d7-84bb-617343832ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs"
Sep 05 19:59:01.935066 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:01.934651    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6pt6\" (UniqueName: \"kubernetes.io/projected/00416952-66bf-46d7-84bb-617343832ad5-kube-api-access-l6pt6\") pod \"payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs\" (UID: \"00416952-66bf-46d7-84bb-617343832ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs"
Sep 05 19:59:01.935066 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:01.934702    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d96c4ef-3daa-42ae-af20-d4640421f102-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-588081-59665c9769-gn8pm\" (UID: \"3d96c4ef-3daa-42ae-af20-d4640421f102\") " pod="openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm"
Sep 05 19:59:01.935066 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:01.934749    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h956b\" (UniqueName: \"kubernetes.io/projected/3d96c4ef-3daa-42ae-af20-d4640421f102-kube-api-access-h956b\") pod \"payload-processing-e2e-inf-a-588081-59665c9769-gn8pm\" (UID: \"3d96c4ef-3daa-42ae-af20-d4640421f102\") " pod="openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm"
Sep 05 19:59:01.935545 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:01.935517    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00416952-66bf-46d7-84bb-617343832ad5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs\" (UID: \"00416952-66bf-46d7-84bb-617343832ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs"
Sep 05 19:59:01.935621 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:01.935605    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d96c4ef-3daa-42ae-af20-d4640421f102-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-588081-59665c9769-gn8pm\" (UID: \"3d96c4ef-3daa-42ae-af20-d4640421f102\") " pod="openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm"
Sep 05 19:59:01.944158 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:01.944070    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h956b\" (UniqueName: \"kubernetes.io/projected/3d96c4ef-3daa-42ae-af20-d4640421f102-kube-api-access-h956b\") pod \"payload-processing-e2e-inf-a-588081-59665c9769-gn8pm\" (UID: \"3d96c4ef-3daa-42ae-af20-d4640421f102\") " pod="openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm"
Sep 05 19:59:01.944899 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:01.944869    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6pt6\" (UniqueName: \"kubernetes.io/projected/00416952-66bf-46d7-84bb-617343832ad5-kube-api-access-l6pt6\") pod \"payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs\" (UID: \"00416952-66bf-46d7-84bb-617343832ad5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs"
Sep 05 19:59:02.027622 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.027576    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs"
Sep 05 19:59:02.061289 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.061256    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm"
Sep 05 19:59:02.081284 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.081258    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7854-140483173004464"
Sep 05 19:59:02.122262 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.122233    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7854-140483173004464_969f4b2e-1a50-4271-948d-0ab2615538b1/test-curl-7854-140483173004464/0.log"
Sep 05 19:59:02.151227 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.148966    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7854-140483173004464"]
Sep 05 19:59:02.154438 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.154394    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7854-140483173004464"]
Sep 05 19:59:02.237764 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.237665    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4mmb\" (UniqueName: \"kubernetes.io/projected/969f4b2e-1a50-4271-948d-0ab2615538b1-kube-api-access-f4mmb\") pod \"969f4b2e-1a50-4271-948d-0ab2615538b1\" (UID: \"969f4b2e-1a50-4271-948d-0ab2615538b1\") "
Sep 05 19:59:02.239973 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.239943    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969f4b2e-1a50-4271-948d-0ab2615538b1-kube-api-access-f4mmb" (OuterVolumeSpecName: "kube-api-access-f4mmb") pod "969f4b2e-1a50-4271-948d-0ab2615538b1" (UID: "969f4b2e-1a50-4271-948d-0ab2615538b1"). InnerVolumeSpecName "kube-api-access-f4mmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:59:02.275724 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.275661    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7854-140483173004464"]
Sep 05 19:59:02.277915 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.276257    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="969f4b2e-1a50-4271-948d-0ab2615538b1" containerName="test-curl-7854-140483173004464"
Sep 05 19:59:02.277915 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.276280    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="969f4b2e-1a50-4271-948d-0ab2615538b1" containerName="test-curl-7854-140483173004464"
Sep 05 19:59:02.277915 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.276365    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="969f4b2e-1a50-4271-948d-0ab2615538b1" containerName="test-curl-7854-140483173004464"
Sep 05 19:59:02.280273 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.280246    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7854-140483173004464"]
Sep 05 19:59:02.280418 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.280355    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7854-140483173004464"
Sep 05 19:59:02.323233 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.323202    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs"]
Sep 05 19:59:02.323764 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:59:02.323739    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00416952_66bf_46d7_84bb_617343832ad5.slice/crio-6f1fbb177512b7961a13e9c63bd230008354336fff9e34a32776f9008e5be63b WatchSource:0}: Error finding container 6f1fbb177512b7961a13e9c63bd230008354336fff9e34a32776f9008e5be63b: Status 404 returned error can't find the container with id 6f1fbb177512b7961a13e9c63bd230008354336fff9e34a32776f9008e5be63b
Sep 05 19:59:02.339180 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.339156    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4mmb\" (UniqueName: \"kubernetes.io/projected/969f4b2e-1a50-4271-948d-0ab2615538b1-kube-api-access-f4mmb\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:59:02.372428 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.372399    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm"]
Sep 05 19:59:02.373456 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:59:02.373419    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d96c4ef_3daa_42ae_af20_d4640421f102.slice/crio-b06c8173ef8deba935948805f0c9dfd039219ccd20edcbf0e1e083dc719cc583 WatchSource:0}: Error finding container b06c8173ef8deba935948805f0c9dfd039219ccd20edcbf0e1e083dc719cc583: Status 404 returned error can't find the container with id b06c8173ef8deba935948805f0c9dfd039219ccd20edcbf0e1e083dc719cc583
Sep 05 19:59:02.439659 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.439635    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qddht\" (UniqueName: \"kubernetes.io/projected/ade0343a-2e92-4a44-a2f6-733b088287a4-kube-api-access-qddht\") pod \"test-curl-7854-140483173004464\" (UID: \"ade0343a-2e92-4a44-a2f6-733b088287a4\") " pod="opendatahub/test-curl-7854-140483173004464"
Sep 05 19:59:02.540509 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.540401    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qddht\" (UniqueName: \"kubernetes.io/projected/ade0343a-2e92-4a44-a2f6-733b088287a4-kube-api-access-qddht\") pod \"test-curl-7854-140483173004464\" (UID: \"ade0343a-2e92-4a44-a2f6-733b088287a4\") " pod="opendatahub/test-curl-7854-140483173004464"
Sep 05 19:59:02.550310 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.550282    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qddht\" (UniqueName: \"kubernetes.io/projected/ade0343a-2e92-4a44-a2f6-733b088287a4-kube-api-access-qddht\") pod \"test-curl-7854-140483173004464\" (UID: \"ade0343a-2e92-4a44-a2f6-733b088287a4\") " pod="opendatahub/test-curl-7854-140483173004464"
Sep 05 19:59:02.590760 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.590724    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7854-140483173004464"
Sep 05 19:59:02.875765 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.875735    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7854-140483173004464"]
Sep 05 19:59:02.877452 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:59:02.877426    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podade0343a_2e92_4a44_a2f6_733b088287a4.slice/crio-837111d2eb61ee4c6d5f95b11addd0b713eb12f9f269ee82149d800604ce0598 WatchSource:0}: Error finding container 837111d2eb61ee4c6d5f95b11addd0b713eb12f9f269ee82149d800604ce0598: Status 404 returned error can't find the container with id 837111d2eb61ee4c6d5f95b11addd0b713eb12f9f269ee82149d800604ce0598
Sep 05 19:59:02.931700 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.931668    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7854-140483173004464"
Sep 05 19:59:02.931854 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.931676    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c45a6459751eea3ed5499591f62b41a9e47aeaa8146e81593e56cb14df858c16"
Sep 05 19:59:02.932965 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.932937    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7854-140483173004464" event={"ID":"ade0343a-2e92-4a44-a2f6-733b088287a4","Type":"ContainerStarted","Data":"837111d2eb61ee4c6d5f95b11addd0b713eb12f9f269ee82149d800604ce0598"}
Sep 05 19:59:02.934501 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.934471    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs" event={"ID":"00416952-66bf-46d7-84bb-617343832ad5","Type":"ContainerStarted","Data":"56a79ffff1a899611ae78f8b3ffaf4e505d5af55ad8f7846649e52c47ca1c682"}
Sep 05 19:59:02.934617 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.934520    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs"
Sep 05 19:59:02.934617 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.934544    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs" event={"ID":"00416952-66bf-46d7-84bb-617343832ad5","Type":"ContainerStarted","Data":"6f1fbb177512b7961a13e9c63bd230008354336fff9e34a32776f9008e5be63b"}
Sep 05 19:59:02.936168 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.936143    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm" event={"ID":"3d96c4ef-3daa-42ae-af20-d4640421f102","Type":"ContainerStarted","Data":"9ea1e8d50270171b8db446041e7279e01e3fdc25ce430bae178571478e495bc0"}
Sep 05 19:59:02.936266 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.936188    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm" event={"ID":"3d96c4ef-3daa-42ae-af20-d4640421f102","Type":"ContainerStarted","Data":"b06c8173ef8deba935948805f0c9dfd039219ccd20edcbf0e1e083dc719cc583"}
Sep 05 19:59:02.936266 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.936213    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm"
Sep 05 19:59:02.955866 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.955800    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs" podStartSLOduration=1.9557801700000002 podStartE2EDuration="1.95578017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:59:02.953076518 +0000 UTC m=+941.982708423" watchObservedRunningTime="2026-09-05 19:59:02.95578017 +0000 UTC m=+941.985412076"
Sep 05 19:59:02.955866 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.955852    2583 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7854-140483173004464" oldPodUID="969f4b2e-1a50-4271-948d-0ab2615538b1" podUID="ade0343a-2e92-4a44-a2f6-733b088287a4"
Sep 05 19:59:02.976607 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:02.976501    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm" podStartSLOduration=1.976481084 podStartE2EDuration="1.976481084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:59:02.974866614 +0000 UTC m=+942.004498519" watchObservedRunningTime="2026-09-05 19:59:02.976481084 +0000 UTC m=+942.006112989"
Sep 05 19:59:03.435127 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:03.435070    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="969f4b2e-1a50-4271-948d-0ab2615538b1" path="/var/lib/kubelet/pods/969f4b2e-1a50-4271-948d-0ab2615538b1/volumes"
Sep 05 19:59:03.943616 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:03.943583    2583 generic.go:356] "Generic (PLEG): container finished" podID="ade0343a-2e92-4a44-a2f6-733b088287a4" containerID="c1eec0843154137e88ffe8ac329f6f21f9dfcb5a83b9404fd56b9578ec7c96f4" exitCode=0
Sep 05 19:59:03.944034 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:03.943667    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7854-140483173004464" event={"ID":"ade0343a-2e92-4a44-a2f6-733b088287a4","Type":"ContainerDied","Data":"c1eec0843154137e88ffe8ac329f6f21f9dfcb5a83b9404fd56b9578ec7c96f4"}
Sep 05 19:59:04.737703 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.737672    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"]
Sep 05 19:59:04.741403 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.741382    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.743617 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.743595    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b4496e2f-openshift-default-dockercfg-s7rm5\""
Sep 05 19:59:04.754160 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.754136    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"]
Sep 05 19:59:04.860859 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.860814    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-workload-certs\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.860859 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.860859    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-istio-data\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.861081 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.860953    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-istio-envoy\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.861081 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.860984    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5af03729-2fae-47c6-a997-87d5d867e133-istio-token\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.861081 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.861005    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5af03729-2fae-47c6-a997-87d5d867e133-istio-podinfo\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.861191 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.861072    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5af03729-2fae-47c6-a997-87d5d867e133-istiod-ca-cert\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.861191 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.861136    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn9pb\" (UniqueName: \"kubernetes.io/projected/5af03729-2fae-47c6-a997-87d5d867e133-kube-api-access-tn9pb\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.861191 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.861185    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-workload-socket\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.861294 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.861207    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-credential-socket\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.962590 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.962553    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5af03729-2fae-47c6-a997-87d5d867e133-istio-podinfo\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.963001 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.962605    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5af03729-2fae-47c6-a997-87d5d867e133-istiod-ca-cert\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.963001 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.962632    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn9pb\" (UniqueName: \"kubernetes.io/projected/5af03729-2fae-47c6-a997-87d5d867e133-kube-api-access-tn9pb\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.963001 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.962829    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-workload-socket\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.963001 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.962881    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-credential-socket\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.963001 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.962995    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-workload-certs\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.963323 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.963025    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-istio-data\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.963323 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.963132    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-istio-envoy\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.963323 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.963159    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5af03729-2fae-47c6-a997-87d5d867e133-istio-token\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.963323 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.963222    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-workload-socket\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.963323 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.963310    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-workload-certs\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.963579 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.963363    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-credential-socket\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.963579 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.963528    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5af03729-2fae-47c6-a997-87d5d867e133-istiod-ca-cert\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.963705 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.963607    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-istio-data\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.969147 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.965701    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-istio-envoy\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.969147 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.965814    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5af03729-2fae-47c6-a997-87d5d867e133-istio-podinfo\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.978701 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.978668    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5af03729-2fae-47c6-a997-87d5d867e133-istio-token\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:04.979483 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:04.979434    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn9pb\" (UniqueName: \"kubernetes.io/projected/5af03729-2fae-47c6-a997-87d5d867e133-kube-api-access-tn9pb\") pod \"e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") " pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:05.051812 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.051775    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:05.089121 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.089080    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7854-140483173004464"
Sep 05 19:59:05.113904 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.112319    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7854-140483173004464_ade0343a-2e92-4a44-a2f6-733b088287a4/test-curl-7854-140483173004464/0.log"
Sep 05 19:59:05.140903 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.138464    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7854-140483173004464"]
Sep 05 19:59:05.145968 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.145923    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7854-140483173004464"]
Sep 05 19:59:05.266893 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.266804    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qddht\" (UniqueName: \"kubernetes.io/projected/ade0343a-2e92-4a44-a2f6-733b088287a4-kube-api-access-qddht\") pod \"ade0343a-2e92-4a44-a2f6-733b088287a4\" (UID: \"ade0343a-2e92-4a44-a2f6-733b088287a4\") "
Sep 05 19:59:05.268903 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.268878    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade0343a-2e92-4a44-a2f6-733b088287a4-kube-api-access-qddht" (OuterVolumeSpecName: "kube-api-access-qddht") pod "ade0343a-2e92-4a44-a2f6-733b088287a4" (UID: "ade0343a-2e92-4a44-a2f6-733b088287a4"). InnerVolumeSpecName "kube-api-access-qddht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:59:05.330651 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.330620    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"]
Sep 05 19:59:05.331611 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:59:05.331585    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af03729_2fae_47c6_a997_87d5d867e133.slice/crio-f1f573ce03aa99388b340400e8116aebae573b72e0a8ba76daf6c666c634120d WatchSource:0}: Error finding container f1f573ce03aa99388b340400e8116aebae573b72e0a8ba76daf6c666c634120d: Status 404 returned error can't find the container with id f1f573ce03aa99388b340400e8116aebae573b72e0a8ba76daf6c666c634120d
Sep 05 19:59:05.333765 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.333735    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:59:05.333879 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.333814    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:59:05.333879 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.333857    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:59:05.368275 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.368245    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qddht\" (UniqueName: \"kubernetes.io/projected/ade0343a-2e92-4a44-a2f6-733b088287a4-kube-api-access-qddht\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:59:05.435409 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.435371    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ade0343a-2e92-4a44-a2f6-733b088287a4" path="/var/lib/kubelet/pods/ade0343a-2e92-4a44-a2f6-733b088287a4/volumes"
Sep 05 19:59:05.885595 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.885561    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"]
Sep 05 19:59:05.885981 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.885967    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade0343a-2e92-4a44-a2f6-733b088287a4" containerName="test-curl-7854-140483173004464"
Sep 05 19:59:05.885981 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.885982    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade0343a-2e92-4a44-a2f6-733b088287a4" containerName="test-curl-7854-140483173004464"
Sep 05 19:59:05.886135 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.886054    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ade0343a-2e92-4a44-a2f6-733b088287a4" containerName="test-curl-7854-140483173004464"
Sep 05 19:59:05.889349 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.889328    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:05.894080 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.894054    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-1fd18b-openshift-default-dockercfg-66f9d\""
Sep 05 19:59:05.901617 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.901595    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"]
Sep 05 19:59:05.955803 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.955772    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7854-140483173004464"
Sep 05 19:59:05.955803 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.955799    2583 scope.go:117] "RemoveContainer" containerID="c1eec0843154137e88ffe8ac329f6f21f9dfcb5a83b9404fd56b9578ec7c96f4"
Sep 05 19:59:05.957635 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.957608    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6" event={"ID":"5af03729-2fae-47c6-a997-87d5d867e133","Type":"ContainerStarted","Data":"96b20ef6b8d4235aa5cbabe6129f5a88a821e6dd1996500c04ae70117af7a0f3"}
Sep 05 19:59:05.957784 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.957664    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6" event={"ID":"5af03729-2fae-47c6-a997-87d5d867e133","Type":"ContainerStarted","Data":"f1f573ce03aa99388b340400e8116aebae573b72e0a8ba76daf6c666c634120d"}
Sep 05 19:59:05.979379 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:05.979337    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6" podStartSLOduration=1.9793220969999998 podStartE2EDuration="1.979322097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:59:05.978355631 +0000 UTC m=+945.007987538" watchObservedRunningTime="2026-09-05 19:59:05.979322097 +0000 UTC m=+945.008953998"
Sep 05 19:59:06.052438 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.052397    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:06.057484 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.057457    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:06.075014 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.074981    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-credential-socket\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.075175 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.075030    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-istio-data\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.075238 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.075215    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b3cd79b-acbb-415f-bda7-43d0d4419620-istio-podinfo\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.075305 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.075258    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b3cd79b-acbb-415f-bda7-43d0d4419620-istio-token\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.075397 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.075369    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-workload-socket\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.075460 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.075410    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-workload-certs\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.075460 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.075431    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b3cd79b-acbb-415f-bda7-43d0d4419620-istiod-ca-cert\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.075568 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.075468    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-istio-envoy\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.075568 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.075484    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jtvg\" (UniqueName: \"kubernetes.io/projected/8b3cd79b-acbb-415f-bda7-43d0d4419620-kube-api-access-7jtvg\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.176261 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.176117    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b3cd79b-acbb-415f-bda7-43d0d4419620-istiod-ca-cert\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.176441 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.176271    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-istio-envoy\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.176441 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.176303    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jtvg\" (UniqueName: \"kubernetes.io/projected/8b3cd79b-acbb-415f-bda7-43d0d4419620-kube-api-access-7jtvg\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.176441 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.176355    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-credential-socket\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.176441 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.176394    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-istio-data\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.176666 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.176465    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b3cd79b-acbb-415f-bda7-43d0d4419620-istio-podinfo\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.176666 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.176509    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b3cd79b-acbb-415f-bda7-43d0d4419620-istio-token\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.176666 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.176583    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-workload-socket\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.176666 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.176618    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-workload-certs\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.176869 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.176773    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-credential-socket\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.176869 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.176849    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-istio-data\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.176966 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.176931    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-workload-socket\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.177028 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.177005    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-workload-certs\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.177357 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.177333    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b3cd79b-acbb-415f-bda7-43d0d4419620-istiod-ca-cert\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.178947 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.178914    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-istio-envoy\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.179048 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.179024    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b3cd79b-acbb-415f-bda7-43d0d4419620-istio-podinfo\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.185446 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.185417    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b3cd79b-acbb-415f-bda7-43d0d4419620-istio-token\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.185550 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.185535    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jtvg\" (UniqueName: \"kubernetes.io/projected/8b3cd79b-acbb-415f-bda7-43d0d4419620-kube-api-access-7jtvg\") pod \"e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") " pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.202507 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.202481    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:06.479813 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.479786    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"]
Sep 05 19:59:06.481706 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:59:06.481680    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3cd79b_acbb_415f_bda7_43d0d4419620.slice/crio-f7d6a4bc66bf97f38fc9fa3a12e555c6b2493ddce9f953e2416e2792fc022c83 WatchSource:0}: Error finding container f7d6a4bc66bf97f38fc9fa3a12e555c6b2493ddce9f953e2416e2792fc022c83: Status 404 returned error can't find the container with id f7d6a4bc66bf97f38fc9fa3a12e555c6b2493ddce9f953e2416e2792fc022c83
Sep 05 19:59:06.484028 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.483992    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:59:06.484136 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.484070    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:59:06.484136 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.484130    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:59:06.965015 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.964973    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs" event={"ID":"8b3cd79b-acbb-415f-bda7-43d0d4419620","Type":"ContainerStarted","Data":"0781b43fded62be76f1965d2bd275a48551dd64b84d96c37e1b0378b448ad02a"}
Sep 05 19:59:06.965222 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.965038    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs" event={"ID":"8b3cd79b-acbb-415f-bda7-43d0d4419620","Type":"ContainerStarted","Data":"f7d6a4bc66bf97f38fc9fa3a12e555c6b2493ddce9f953e2416e2792fc022c83"}
Sep 05 19:59:06.967421 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.967391    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:06.968337 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.968317    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:06.988308 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:06.988262    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs" podStartSLOduration=1.988246979 podStartE2EDuration="1.988246979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:59:06.985365924 +0000 UTC m=+946.014997830" watchObservedRunningTime="2026-09-05 19:59:06.988246979 +0000 UTC m=+946.017878881"
Sep 05 19:59:07.203253 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:07.203219    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:07.208937 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:07.208912    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:07.977883 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:07.977848    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:07.978805 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:07.978780    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:09.221959 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:09.221925    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl"]
Sep 05 19:59:09.225730 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:09.225708    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl"
Sep 05 19:59:09.227844 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:09.227822    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b4496e2f\""
Sep 05 19:59:09.237794 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:09.237762    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl"]
Sep 05 19:59:09.312790 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:09.312756    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pkzr\" (UniqueName: \"kubernetes.io/projected/e340f762-a090-4588-80bc-4cdf039d1c0b-kube-api-access-6pkzr\") pod \"payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl\" (UID: \"e340f762-a090-4588-80bc-4cdf039d1c0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl"
Sep 05 19:59:09.312983 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:09.312806    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e340f762-a090-4588-80bc-4cdf039d1c0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl\" (UID: \"e340f762-a090-4588-80bc-4cdf039d1c0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl"
Sep 05 19:59:09.413912 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:09.413872    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e340f762-a090-4588-80bc-4cdf039d1c0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl\" (UID: \"e340f762-a090-4588-80bc-4cdf039d1c0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl"
Sep 05 19:59:09.414188 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:09.414039    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pkzr\" (UniqueName: \"kubernetes.io/projected/e340f762-a090-4588-80bc-4cdf039d1c0b-kube-api-access-6pkzr\") pod \"payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl\" (UID: \"e340f762-a090-4588-80bc-4cdf039d1c0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl"
Sep 05 19:59:09.414871 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:09.414845    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e340f762-a090-4588-80bc-4cdf039d1c0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl\" (UID: \"e340f762-a090-4588-80bc-4cdf039d1c0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl"
Sep 05 19:59:09.426424 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:09.426400    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pkzr\" (UniqueName: \"kubernetes.io/projected/e340f762-a090-4588-80bc-4cdf039d1c0b-kube-api-access-6pkzr\") pod \"payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl\" (UID: \"e340f762-a090-4588-80bc-4cdf039d1c0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl"
Sep 05 19:59:09.540101 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:09.540014    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl"
Sep 05 19:59:09.818052 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:09.818016    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl"]
Sep 05 19:59:09.818813 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:59:09.818784    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode340f762_a090_4588_80bc_4cdf039d1c0b.slice/crio-d47da00694341060867f4923de48e6ca1134d5089b059d01aa1a78dbe2ce57b7 WatchSource:0}: Error finding container d47da00694341060867f4923de48e6ca1134d5089b059d01aa1a78dbe2ce57b7: Status 404 returned error can't find the container with id d47da00694341060867f4923de48e6ca1134d5089b059d01aa1a78dbe2ce57b7
Sep 05 19:59:09.994960 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:09.994920    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl" event={"ID":"e340f762-a090-4588-80bc-4cdf039d1c0b","Type":"ContainerStarted","Data":"7524644dc45b24f261e5b80c384b660065111c1e097e04f55bdeb914ec653f38"}
Sep 05 19:59:09.995151 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:09.994984    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl" event={"ID":"e340f762-a090-4588-80bc-4cdf039d1c0b","Type":"ContainerStarted","Data":"d47da00694341060867f4923de48e6ca1134d5089b059d01aa1a78dbe2ce57b7"}
Sep 05 19:59:09.995151 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:09.995014    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl"
Sep 05 19:59:10.014624 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:10.014566    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl" podStartSLOduration=1.014549847 podStartE2EDuration="1.014549847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:59:10.012340312 +0000 UTC m=+949.041972221" watchObservedRunningTime="2026-09-05 19:59:10.014549847 +0000 UTC m=+949.044181752"
Sep 05 19:59:10.945042 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:10.945007    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb"]
Sep 05 19:59:10.948471 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:10.948455    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb"
Sep 05 19:59:10.950804 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:10.950775    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-1fd18b\""
Sep 05 19:59:10.950804 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:10.950782    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-1fd18b-dockercfg-wmkbr\""
Sep 05 19:59:10.958540 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:10.958512    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb"]
Sep 05 19:59:11.032620 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:11.032581    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-296f5\" (UniqueName: \"kubernetes.io/projected/7c2bab11-ce91-4eea-9a5d-47f8aff048f3-kube-api-access-296f5\") pod \"payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb\" (UID: \"7c2bab11-ce91-4eea-9a5d-47f8aff048f3\") " pod="openshift-ingress/payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb"
Sep 05 19:59:11.033043 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:11.033010    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c2bab11-ce91-4eea-9a5d-47f8aff048f3-plugins-config-volume\") pod \"payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb\" (UID: \"7c2bab11-ce91-4eea-9a5d-47f8aff048f3\") " pod="openshift-ingress/payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb"
Sep 05 19:59:11.133680 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:11.133646    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-296f5\" (UniqueName: \"kubernetes.io/projected/7c2bab11-ce91-4eea-9a5d-47f8aff048f3-kube-api-access-296f5\") pod \"payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb\" (UID: \"7c2bab11-ce91-4eea-9a5d-47f8aff048f3\") " pod="openshift-ingress/payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb"
Sep 05 19:59:11.133842 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:11.133756    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c2bab11-ce91-4eea-9a5d-47f8aff048f3-plugins-config-volume\") pod \"payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb\" (UID: \"7c2bab11-ce91-4eea-9a5d-47f8aff048f3\") " pod="openshift-ingress/payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb"
Sep 05 19:59:11.134471 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:11.134451    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c2bab11-ce91-4eea-9a5d-47f8aff048f3-plugins-config-volume\") pod \"payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb\" (UID: \"7c2bab11-ce91-4eea-9a5d-47f8aff048f3\") " pod="openshift-ingress/payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb"
Sep 05 19:59:11.142165 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:11.142133    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-296f5\" (UniqueName: \"kubernetes.io/projected/7c2bab11-ce91-4eea-9a5d-47f8aff048f3-kube-api-access-296f5\") pod \"payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb\" (UID: \"7c2bab11-ce91-4eea-9a5d-47f8aff048f3\") " pod="openshift-ingress/payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb"
Sep 05 19:59:11.259868 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:11.259781    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb"
Sep 05 19:59:11.535847 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:11.535812    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb"]
Sep 05 19:59:11.536925 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:59:11.536894    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2bab11_ce91_4eea_9a5d_47f8aff048f3.slice/crio-ed4d9319deba39dd3f1868034060d8b2cea98ac500881262c8be5a0e2225ed13 WatchSource:0}: Error finding container ed4d9319deba39dd3f1868034060d8b2cea98ac500881262c8be5a0e2225ed13: Status 404 returned error can't find the container with id ed4d9319deba39dd3f1868034060d8b2cea98ac500881262c8be5a0e2225ed13
Sep 05 19:59:12.014735 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:12.014694    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb" event={"ID":"7c2bab11-ce91-4eea-9a5d-47f8aff048f3","Type":"ContainerStarted","Data":"acb1ca7e382c6d46092935bbc3247da49ee9c5ca3a2228d06ec1cbed2a01b669"}
Sep 05 19:59:12.015137 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:12.014757    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb" event={"ID":"7c2bab11-ce91-4eea-9a5d-47f8aff048f3","Type":"ContainerStarted","Data":"ed4d9319deba39dd3f1868034060d8b2cea98ac500881262c8be5a0e2225ed13"}
Sep 05 19:59:12.015137 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:12.014857    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb"
Sep 05 19:59:12.031850 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:12.031799    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb" podStartSLOduration=2.031780262 podStartE2EDuration="2.031780262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:59:12.031175575 +0000 UTC m=+951.060807482" watchObservedRunningTime="2026-09-05 19:59:12.031780262 +0000 UTC m=+951.061412170"
Sep 05 19:59:13.945514 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:13.945481    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm"
Sep 05 19:59:13.945942 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:13.945553    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs"
Sep 05 19:59:21.010046 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:21.009999    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl"
Sep 05 19:59:23.022768 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:23.022727    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb"
Sep 05 19:59:24.019705 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:24.019283    2583 patch_prober.go:28] interesting pod/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 05 19:59:24.019705 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:24.019357    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs" podUID="8b3cd79b-acbb-415f-bda7-43d0d4419620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 05 19:59:28.133885 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:28.133852    2583 generic.go:356] "Generic (PLEG): container finished" podID="56770103-b7ef-4c4c-8868-03e81ec44ad5" containerID="00464ff44930788a5c94267fc00754c3a4d59947c73d329267cd032ecdba962d" exitCode=0
Sep 05 19:59:28.134301 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:28.133927    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66" event={"ID":"56770103-b7ef-4c4c-8868-03e81ec44ad5","Type":"ContainerDied","Data":"00464ff44930788a5c94267fc00754c3a4d59947c73d329267cd032ecdba962d"}
Sep 05 19:59:28.134397 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:28.134377    2583 scope.go:117] "RemoveContainer" containerID="00464ff44930788a5c94267fc00754c3a4d59947c73d329267cd032ecdba962d"
Sep 05 19:59:28.135572 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:28.135544    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:59:28.135692 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:28.135628    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:59:28.135692 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:28.135675    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:59:28.135811 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:28.135718    2583 generic.go:356] "Generic (PLEG): container finished" podID="5af03729-2fae-47c6-a997-87d5d867e133" containerID="96b20ef6b8d4235aa5cbabe6129f5a88a821e6dd1996500c04ae70117af7a0f3" exitCode=0
Sep 05 19:59:28.135811 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:28.135786    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6" event={"ID":"5af03729-2fae-47c6-a997-87d5d867e133","Type":"ContainerDied","Data":"96b20ef6b8d4235aa5cbabe6129f5a88a821e6dd1996500c04ae70117af7a0f3"}
Sep 05 19:59:28.136096 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:28.136083    2583 scope.go:117] "RemoveContainer" containerID="96b20ef6b8d4235aa5cbabe6129f5a88a821e6dd1996500c04ae70117af7a0f3"
Sep 05 19:59:28.137057 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:28.137033    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:59:28.137130 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:28.137088    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:59:28.137178 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:28.137135    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:59:28.903977 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:28.903934    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:59:29.143034 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:29.142996    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6" event={"ID":"5af03729-2fae-47c6-a997-87d5d867e133","Type":"ContainerStarted","Data":"2deb0fa2238b4bc1d57ca351206a9b6a590de57e3eb0357aba4f9468bdc25367"}
Sep 05 19:59:29.145248 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:29.145225    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66" event={"ID":"56770103-b7ef-4c4c-8868-03e81ec44ad5","Type":"ContainerStarted","Data":"9066fedb0e16a843eabd235b7028f4bce73a344d2b8d8e4f664bfffc407fc26c"}
Sep 05 19:59:29.146861 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:29.146839    2583 generic.go:356] "Generic (PLEG): container finished" podID="8b3cd79b-acbb-415f-bda7-43d0d4419620" containerID="0781b43fded62be76f1965d2bd275a48551dd64b84d96c37e1b0378b448ad02a" exitCode=0
Sep 05 19:59:29.146948 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:29.146866    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs" event={"ID":"8b3cd79b-acbb-415f-bda7-43d0d4419620","Type":"ContainerDied","Data":"0781b43fded62be76f1965d2bd275a48551dd64b84d96c37e1b0378b448ad02a"}
Sep 05 19:59:29.147195 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:29.147179    2583 scope.go:117] "RemoveContainer" containerID="0781b43fded62be76f1965d2bd275a48551dd64b84d96c37e1b0378b448ad02a"
Sep 05 19:59:29.148194 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:29.148168    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:59:29.148257 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:29.148232    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:59:29.148296 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:29.148263    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 19:59:30.004942 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:30.004857    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:59:30.004942 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:30.004926    2583 patch_prober.go:28] interesting pod/e2e-inf-a-588081-openshift-default-648bd9598-kmp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 05 19:59:30.005235 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:30.004983    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66" podUID="56770103-b7ef-4c4c-8868-03e81ec44ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 05 19:59:30.052901 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:30.052867    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:30.053100 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:30.053078    2583 patch_prober.go:28] interesting pod/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 05 19:59:30.053170 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:30.053149    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6" podUID="5af03729-2fae-47c6-a997-87d5d867e133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 05 19:59:30.155535 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:30.155463    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs" event={"ID":"8b3cd79b-acbb-415f-bda7-43d0d4419620","Type":"ContainerStarted","Data":"1d71e43a9a81717091e294cd35c779cb2fe804761d212a8666764da132649e66"}
Sep 05 19:59:30.204557 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:30.204523    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:30.204947 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:30.204925    2583 patch_prober.go:28] interesting pod/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 05 19:59:30.205190 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:30.205161    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs" podUID="8b3cd79b-acbb-415f-bda7-43d0d4419620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 05 19:59:31.004128 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:31.003584    2583 patch_prober.go:28] interesting pod/e2e-inf-a-588081-openshift-default-648bd9598-kmp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 05 19:59:31.004128 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:31.003670    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66" podUID="56770103-b7ef-4c4c-8868-03e81ec44ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 05 19:59:31.052822 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:31.052781    2583 patch_prober.go:28] interesting pod/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 05 19:59:31.052990 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:31.052839    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6" podUID="5af03729-2fae-47c6-a997-87d5d867e133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 05 19:59:31.203585 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:31.203548    2583 patch_prober.go:28] interesting pod/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 05 19:59:31.204185 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:31.204153    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs" podUID="8b3cd79b-acbb-415f-bda7-43d0d4419620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 05 19:59:32.004496 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:32.004454    2583 patch_prober.go:28] interesting pod/e2e-inf-a-588081-openshift-default-648bd9598-kmp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 05 19:59:32.004705 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:32.004521    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66" podUID="56770103-b7ef-4c4c-8868-03e81ec44ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 05 19:59:32.053181 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:32.053139    2583 patch_prober.go:28] interesting pod/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 05 19:59:32.053353 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:32.053191    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6" podUID="5af03729-2fae-47c6-a997-87d5d867e133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 05 19:59:32.203412 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:32.203375    2583 patch_prober.go:28] interesting pod/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 05 19:59:32.204079 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:32.203678    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs" podUID="8b3cd79b-acbb-415f-bda7-43d0d4419620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 05 19:59:32.769734 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:32.769679    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4496e2f-jbb4w"]
Sep 05 19:59:32.774093 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:32.774059    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4496e2f-jbb4w"
Sep 05 19:59:32.776643 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:32.776592    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zx89\""
Sep 05 19:59:32.777153 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:32.777131    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 19:59:32.781151 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:32.781124    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4496e2f-jbb4w"]
Sep 05 19:59:32.893133 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:32.889593    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39a29887-ac3c-4ac9-9760-79e5bc05b9e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b4496e2f-jbb4w\" (UID: \"39a29887-ac3c-4ac9-9760-79e5bc05b9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4496e2f-jbb4w"
Sep 05 19:59:33.022186 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.021952    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39a29887-ac3c-4ac9-9760-79e5bc05b9e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b4496e2f-jbb4w\" (UID: \"39a29887-ac3c-4ac9-9760-79e5bc05b9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4496e2f-jbb4w"
Sep 05 19:59:33.022186 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.022017    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39a29887-ac3c-4ac9-9760-79e5bc05b9e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b4496e2f-jbb4w\" (UID: \"39a29887-ac3c-4ac9-9760-79e5bc05b9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4496e2f-jbb4w"
Sep 05 19:59:33.030132 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.028362    2583 patch_prober.go:28] interesting pod/e2e-inf-a-588081-openshift-default-648bd9598-kmp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 05 19:59:33.030132 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.028431    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66" podUID="56770103-b7ef-4c4c-8868-03e81ec44ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 05 19:59:33.053133 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.052522    2583 patch_prober.go:28] interesting pod/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 05 19:59:33.053133 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.052580    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6" podUID="5af03729-2fae-47c6-a997-87d5d867e133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 05 19:59:33.095131 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.094796    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw"]
Sep 05 19:59:33.103212 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.095978    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4496e2f-jbb4w"
Sep 05 19:59:33.106688 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.106634    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw"
Sep 05 19:59:33.128632 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.128604    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1dc7b1\""
Sep 05 19:59:33.219323 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.219284    2583 patch_prober.go:28] interesting pod/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 05 19:59:33.219853 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.219819    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs" podUID="8b3cd79b-acbb-415f-bda7-43d0d4419620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 05 19:59:33.238085 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.238049    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw"]
Sep 05 19:59:33.257814 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.257772    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw\" (UID: \"87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw"
Sep 05 19:59:33.258189 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.258146    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x22hq\" (UniqueName: \"kubernetes.io/projected/87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa-kube-api-access-x22hq\") pod \"payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw\" (UID: \"87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw"
Sep 05 19:59:33.289276 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.289186    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9"]
Sep 05 19:59:33.332470 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.332430    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9"]
Sep 05 19:59:33.332651 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.332601    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9"
Sep 05 19:59:33.342135 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.340399    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1dc7b1-dockercfg-scf7g\""
Sep 05 19:59:33.359612 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.359578    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw\" (UID: \"87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw"
Sep 05 19:59:33.359869 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.359847    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x22hq\" (UniqueName: \"kubernetes.io/projected/87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa-kube-api-access-x22hq\") pod \"payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw\" (UID: \"87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw"
Sep 05 19:59:33.359999 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.359982    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkrr6\" (UniqueName: \"kubernetes.io/projected/22b964ab-a361-4c66-af33-03426c04b2e6-kube-api-access-kkrr6\") pod \"payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9\" (UID: \"22b964ab-a361-4c66-af33-03426c04b2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9"
Sep 05 19:59:33.360201 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.360180    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b964ab-a361-4c66-af33-03426c04b2e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9\" (UID: \"22b964ab-a361-4c66-af33-03426c04b2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9"
Sep 05 19:59:33.361250 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.361228    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw\" (UID: \"87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw"
Sep 05 19:59:33.379777 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.379746    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x22hq\" (UniqueName: \"kubernetes.io/projected/87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa-kube-api-access-x22hq\") pod \"payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw\" (UID: \"87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw"
Sep 05 19:59:33.461866 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.461835    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkrr6\" (UniqueName: \"kubernetes.io/projected/22b964ab-a361-4c66-af33-03426c04b2e6-kube-api-access-kkrr6\") pod \"payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9\" (UID: \"22b964ab-a361-4c66-af33-03426c04b2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9"
Sep 05 19:59:33.462142 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.462099    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b964ab-a361-4c66-af33-03426c04b2e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9\" (UID: \"22b964ab-a361-4c66-af33-03426c04b2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9"
Sep 05 19:59:33.463086 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.463064    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b964ab-a361-4c66-af33-03426c04b2e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9\" (UID: \"22b964ab-a361-4c66-af33-03426c04b2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9"
Sep 05 19:59:33.489212 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.489167    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkrr6\" (UniqueName: \"kubernetes.io/projected/22b964ab-a361-4c66-af33-03426c04b2e6-kube-api-access-kkrr6\") pod \"payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9\" (UID: \"22b964ab-a361-4c66-af33-03426c04b2e6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9"
Sep 05 19:59:33.534263 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.533960    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw"
Sep 05 19:59:33.685132 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.679290    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9"
Sep 05 19:59:33.751151 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:33.751089    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4496e2f-jbb4w"]
Sep 05 19:59:34.005932 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:34.005488    2583 patch_prober.go:28] interesting pod/e2e-inf-a-588081-openshift-default-648bd9598-kmp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 05 19:59:34.005932 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:34.005547    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66" podUID="56770103-b7ef-4c4c-8868-03e81ec44ad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 05 19:59:34.065136 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:34.065040    2583 patch_prober.go:28] interesting pod/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 05 19:59:34.077138 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:34.069182    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6" podUID="5af03729-2fae-47c6-a997-87d5d867e133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 05 19:59:34.173192 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:34.172989    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw"]
Sep 05 19:59:34.202407 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:59:34.202363    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a8f5b6_5861_49cb_81d2_8b71cdbc1eaa.slice/crio-f524cbd7580c2e03c1b868192e5be7810b48e476245d357078177371be4e833f WatchSource:0}: Error finding container f524cbd7580c2e03c1b868192e5be7810b48e476245d357078177371be4e833f: Status 404 returned error can't find the container with id f524cbd7580c2e03c1b868192e5be7810b48e476245d357078177371be4e833f
Sep 05 19:59:34.213321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:34.210222    2583 patch_prober.go:28] interesting pod/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 05 19:59:34.213321 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:34.210282    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs" podUID="8b3cd79b-acbb-415f-bda7-43d0d4419620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 05 19:59:34.249903 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:34.249784    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9"]
Sep 05 19:59:34.292542 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:34.291103    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw" event={"ID":"87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa","Type":"ContainerStarted","Data":"f524cbd7580c2e03c1b868192e5be7810b48e476245d357078177371be4e833f"}
Sep 05 19:59:34.296671 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:34.296590    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4496e2f-jbb4w" event={"ID":"39a29887-ac3c-4ac9-9760-79e5bc05b9e0","Type":"ContainerStarted","Data":"139db66cdfee6613833633ad0558b6ad5e6f35ff45aeb6a22ec89295027bc61c"}
Sep 05 19:59:35.017009 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:35.014387    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:59:35.017009 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:35.015649    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:59:35.022561 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:35.022334    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 19:59:35.157625 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:35.157583    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:35.158417 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:35.158395    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:35.159595 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:35.159515    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:35.205753 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:35.205722    2583 patch_prober.go:28] interesting pod/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 05 19:59:35.206050 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:35.206018    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs" podUID="8b3cd79b-acbb-415f-bda7-43d0d4419620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 05 19:59:35.316887 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:35.315912    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9" event={"ID":"22b964ab-a361-4c66-af33-03426c04b2e6","Type":"ContainerStarted","Data":"fbd3ca49f64be86b230e8314629c208bf06d578981db8a3976d930588a1c1f1a"}
Sep 05 19:59:35.316887 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:35.315979    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9" event={"ID":"22b964ab-a361-4c66-af33-03426c04b2e6","Type":"ContainerStarted","Data":"967380b26ccac0e8e32781fe441080917ed1b97f52786f95d5abacbf03432106"}
Sep 05 19:59:35.316887 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:35.316233    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9"
Sep 05 19:59:35.319449 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:35.318733    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw" event={"ID":"87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa","Type":"ContainerStarted","Data":"452c84c07df8fe8ea71d50a609330e113dab908becea739d1b98b5cbf231fc3f"}
Sep 05 19:59:35.335281 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:35.335229    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9" podStartSLOduration=2.335211036 podStartE2EDuration="2.335211036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:59:35.332021632 +0000 UTC m=+974.361653540" watchObservedRunningTime="2026-09-05 19:59:35.335211036 +0000 UTC m=+974.364842941"
Sep 05 19:59:35.351036 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:35.350994    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw" podStartSLOduration=2.350978479 podStartE2EDuration="2.350978479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:59:35.347651405 +0000 UTC m=+974.377283312" watchObservedRunningTime="2026-09-05 19:59:35.350978479 +0000 UTC m=+974.380610378"
Sep 05 19:59:36.203566 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:36.203488    2583 patch_prober.go:28] interesting pod/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 05 19:59:36.203566 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:36.203561    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs" podUID="8b3cd79b-acbb-415f-bda7-43d0d4419620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 05 19:59:36.327920 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:36.327725    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw"
Sep 05 19:59:37.208286 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:37.208247    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:37.208692 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:37.208664    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:37.209361 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:37.209338    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 19:59:38.341746 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:38.341708    2583 generic.go:356] "Generic (PLEG): container finished" podID="39a29887-ac3c-4ac9-9760-79e5bc05b9e0" containerID="38e5434007fe48e999ce81b9473a881b5a254cb7cd320b50427ebd8eb8e58aeb" exitCode=0
Sep 05 19:59:38.342146 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:38.341797    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4496e2f-jbb4w" event={"ID":"39a29887-ac3c-4ac9-9760-79e5bc05b9e0","Type":"ContainerDied","Data":"38e5434007fe48e999ce81b9473a881b5a254cb7cd320b50427ebd8eb8e58aeb"}
Sep 05 19:59:39.493201 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:39.493170    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4496e2f-jbb4w"
Sep 05 19:59:39.571124 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:39.571078    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39a29887-ac3c-4ac9-9760-79e5bc05b9e0-maas-api-service-ca\") pod \"39a29887-ac3c-4ac9-9760-79e5bc05b9e0\" (UID: \"39a29887-ac3c-4ac9-9760-79e5bc05b9e0\") "
Sep 05 19:59:39.571450 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:39.571425    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39a29887-ac3c-4ac9-9760-79e5bc05b9e0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "39a29887-ac3c-4ac9-9760-79e5bc05b9e0" (UID: "39a29887-ac3c-4ac9-9760-79e5bc05b9e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 19:59:39.672187 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:39.672096    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39a29887-ac3c-4ac9-9760-79e5bc05b9e0-maas-api-service-ca\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:59:40.355478 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:40.355440    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4496e2f-jbb4w"
Sep 05 19:59:40.355684 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:40.355443    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4496e2f-jbb4w" event={"ID":"39a29887-ac3c-4ac9-9760-79e5bc05b9e0","Type":"ContainerDied","Data":"139db66cdfee6613833633ad0558b6ad5e6f35ff45aeb6a22ec89295027bc61c"}
Sep 05 19:59:40.355684 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:40.355587    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="139db66cdfee6613833633ad0558b6ad5e6f35ff45aeb6a22ec89295027bc61c"
Sep 05 19:59:46.330517 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:46.330433    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9"
Sep 05 19:59:47.337037 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:47.337008    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw"
Sep 05 19:59:49.286445 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:49.286408    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl"]
Sep 05 19:59:49.286919 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:49.286707    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl" podUID="e340f762-a090-4588-80bc-4cdf039d1c0b" containerName="payload-pre-processing" containerID="cri-o://7524644dc45b24f261e5b80c384b660065111c1e097e04f55bdeb914ec653f38" gracePeriod=30
Sep 05 19:59:49.418325 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:49.418289    2583 generic.go:356] "Generic (PLEG): container finished" podID="e340f762-a090-4588-80bc-4cdf039d1c0b" containerID="7524644dc45b24f261e5b80c384b660065111c1e097e04f55bdeb914ec653f38" exitCode=0
Sep 05 19:59:49.418458 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:49.418361    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl" event={"ID":"e340f762-a090-4588-80bc-4cdf039d1c0b","Type":"ContainerDied","Data":"7524644dc45b24f261e5b80c384b660065111c1e097e04f55bdeb914ec653f38"}
Sep 05 19:59:49.444996 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:49.444962    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4496e2f-jbb4w"]
Sep 05 19:59:49.447565 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:49.447536    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b4496e2f-jbb4w"]
Sep 05 19:59:49.572844 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:49.568089    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"]
Sep 05 19:59:49.572844 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:49.568349    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6" podUID="5af03729-2fae-47c6-a997-87d5d867e133" containerName="istio-proxy" containerID="cri-o://2deb0fa2238b4bc1d57ca351206a9b6a590de57e3eb0357aba4f9468bdc25367" gracePeriod=30
Sep 05 19:59:49.595484 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:49.595454    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl"
Sep 05 19:59:49.673931 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:49.673899    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pkzr\" (UniqueName: \"kubernetes.io/projected/e340f762-a090-4588-80bc-4cdf039d1c0b-kube-api-access-6pkzr\") pod \"e340f762-a090-4588-80bc-4cdf039d1c0b\" (UID: \"e340f762-a090-4588-80bc-4cdf039d1c0b\") "
Sep 05 19:59:49.674188 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:49.673984    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e340f762-a090-4588-80bc-4cdf039d1c0b-plugins-config-volume\") pod \"e340f762-a090-4588-80bc-4cdf039d1c0b\" (UID: \"e340f762-a090-4588-80bc-4cdf039d1c0b\") "
Sep 05 19:59:49.674453 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:49.674429    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e340f762-a090-4588-80bc-4cdf039d1c0b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e340f762-a090-4588-80bc-4cdf039d1c0b" (UID: "e340f762-a090-4588-80bc-4cdf039d1c0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 19:59:49.676187 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:49.676159    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e340f762-a090-4588-80bc-4cdf039d1c0b-kube-api-access-6pkzr" (OuterVolumeSpecName: "kube-api-access-6pkzr") pod "e340f762-a090-4588-80bc-4cdf039d1c0b" (UID: "e340f762-a090-4588-80bc-4cdf039d1c0b"). InnerVolumeSpecName "kube-api-access-6pkzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:59:49.775485 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:49.775453    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pkzr\" (UniqueName: \"kubernetes.io/projected/e340f762-a090-4588-80bc-4cdf039d1c0b-kube-api-access-6pkzr\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:59:49.775485 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:49.775482    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e340f762-a090-4588-80bc-4cdf039d1c0b-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:59:50.160482 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:50.160444    2583 patch_prober.go:28] interesting pod/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 05 19:59:50.160648 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:50.160499    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6" podUID="5af03729-2fae-47c6-a997-87d5d867e133" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 19:59:50.427505 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:50.427427    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl"
Sep 05 19:59:50.427505 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:50.427450    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl" event={"ID":"e340f762-a090-4588-80bc-4cdf039d1c0b","Type":"ContainerDied","Data":"d47da00694341060867f4923de48e6ca1134d5089b059d01aa1a78dbe2ce57b7"}
Sep 05 19:59:50.427940 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:50.427511    2583 scope.go:117] "RemoveContainer" containerID="7524644dc45b24f261e5b80c384b660065111c1e097e04f55bdeb914ec653f38"
Sep 05 19:59:50.453129 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:50.452311    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl"]
Sep 05 19:59:50.457858 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:50.457823    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b4496e2f-54c4f7c57d-2xnpl"]
Sep 05 19:59:51.435278 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:51.435242    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39a29887-ac3c-4ac9-9760-79e5bc05b9e0" path="/var/lib/kubelet/pods/39a29887-ac3c-4ac9-9760-79e5bc05b9e0/volumes"
Sep 05 19:59:51.435646 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:51.435631    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e340f762-a090-4588-80bc-4cdf039d1c0b" path="/var/lib/kubelet/pods/e340f762-a090-4588-80bc-4cdf039d1c0b/volumes"
Sep 05 19:59:54.835520 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:54.835493    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:54.925518 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:54.925437    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn9pb\" (UniqueName: \"kubernetes.io/projected/5af03729-2fae-47c6-a997-87d5d867e133-kube-api-access-tn9pb\") pod \"5af03729-2fae-47c6-a997-87d5d867e133\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") "
Sep 05 19:59:54.925518 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:54.925487    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-istio-data\") pod \"5af03729-2fae-47c6-a997-87d5d867e133\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") "
Sep 05 19:59:54.925518 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:54.925510    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5af03729-2fae-47c6-a997-87d5d867e133-istio-podinfo\") pod \"5af03729-2fae-47c6-a997-87d5d867e133\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") "
Sep 05 19:59:54.925787 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:54.925524    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-workload-socket\") pod \"5af03729-2fae-47c6-a997-87d5d867e133\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") "
Sep 05 19:59:54.925787 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:54.925575    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-credential-socket\") pod \"5af03729-2fae-47c6-a997-87d5d867e133\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") "
Sep 05 19:59:54.925787 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:54.925593    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5af03729-2fae-47c6-a997-87d5d867e133-istiod-ca-cert\") pod \"5af03729-2fae-47c6-a997-87d5d867e133\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") "
Sep 05 19:59:54.925787 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:54.925639    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-workload-certs\") pod \"5af03729-2fae-47c6-a997-87d5d867e133\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") "
Sep 05 19:59:54.925787 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:54.925675    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5af03729-2fae-47c6-a997-87d5d867e133-istio-token\") pod \"5af03729-2fae-47c6-a997-87d5d867e133\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") "
Sep 05 19:59:54.925787 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:54.925702    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-istio-envoy\") pod \"5af03729-2fae-47c6-a997-87d5d867e133\" (UID: \"5af03729-2fae-47c6-a997-87d5d867e133\") "
Sep 05 19:59:54.925787 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:54.925752    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-istio-data" (OuterVolumeSpecName: "istio-data") pod "5af03729-2fae-47c6-a997-87d5d867e133" (UID: "5af03729-2fae-47c6-a997-87d5d867e133"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:59:54.926624 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:54.926316    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5af03729-2fae-47c6-a997-87d5d867e133" (UID: "5af03729-2fae-47c6-a997-87d5d867e133"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:59:54.926624 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:54.926387    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5af03729-2fae-47c6-a997-87d5d867e133" (UID: "5af03729-2fae-47c6-a997-87d5d867e133"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:59:54.926624 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:54.926377    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5af03729-2fae-47c6-a997-87d5d867e133" (UID: "5af03729-2fae-47c6-a997-87d5d867e133"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:59:54.926624 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:54.926560    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-istio-data\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:59:54.926902 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:54.926690    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af03729-2fae-47c6-a997-87d5d867e133-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5af03729-2fae-47c6-a997-87d5d867e133" (UID: "5af03729-2fae-47c6-a997-87d5d867e133"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 19:59:54.933153 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:54.928343    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af03729-2fae-47c6-a997-87d5d867e133-kube-api-access-tn9pb" (OuterVolumeSpecName: "kube-api-access-tn9pb") pod "5af03729-2fae-47c6-a997-87d5d867e133" (UID: "5af03729-2fae-47c6-a997-87d5d867e133"). InnerVolumeSpecName "kube-api-access-tn9pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:59:54.933153 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:54.928654    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5af03729-2fae-47c6-a997-87d5d867e133-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5af03729-2fae-47c6-a997-87d5d867e133" (UID: "5af03729-2fae-47c6-a997-87d5d867e133"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 19:59:54.933153 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:54.928946    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5af03729-2fae-47c6-a997-87d5d867e133" (UID: "5af03729-2fae-47c6-a997-87d5d867e133"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 19:59:54.933153 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:54.930251    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af03729-2fae-47c6-a997-87d5d867e133-istio-token" (OuterVolumeSpecName: "istio-token") pod "5af03729-2fae-47c6-a997-87d5d867e133" (UID: "5af03729-2fae-47c6-a997-87d5d867e133"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 19:59:55.027658 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:55.027621    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn9pb\" (UniqueName: \"kubernetes.io/projected/5af03729-2fae-47c6-a997-87d5d867e133-kube-api-access-tn9pb\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:59:55.027658 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:55.027653    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5af03729-2fae-47c6-a997-87d5d867e133-istio-podinfo\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:59:55.027658 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:55.027663    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-workload-socket\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:59:55.027886 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:55.027672    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-credential-socket\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:59:55.027886 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:55.027682    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5af03729-2fae-47c6-a997-87d5d867e133-istiod-ca-cert\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:59:55.027886 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:55.027690    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-workload-certs\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:59:55.027886 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:55.027698    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5af03729-2fae-47c6-a997-87d5d867e133-istio-token\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:59:55.027886 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:55.027705    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5af03729-2fae-47c6-a997-87d5d867e133-istio-envoy\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 19:59:55.462560 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:55.462528    2583 generic.go:356] "Generic (PLEG): container finished" podID="5af03729-2fae-47c6-a997-87d5d867e133" containerID="2deb0fa2238b4bc1d57ca351206a9b6a590de57e3eb0357aba4f9468bdc25367" exitCode=0
Sep 05 19:59:55.462701 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:55.462594    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"
Sep 05 19:59:55.462701 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:55.462606    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6" event={"ID":"5af03729-2fae-47c6-a997-87d5d867e133","Type":"ContainerDied","Data":"2deb0fa2238b4bc1d57ca351206a9b6a590de57e3eb0357aba4f9468bdc25367"}
Sep 05 19:59:55.462701 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:55.462664    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6" event={"ID":"5af03729-2fae-47c6-a997-87d5d867e133","Type":"ContainerDied","Data":"f1f573ce03aa99388b340400e8116aebae573b72e0a8ba76daf6c666c634120d"}
Sep 05 19:59:55.462701 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:55.462688    2583 scope.go:117] "RemoveContainer" containerID="2deb0fa2238b4bc1d57ca351206a9b6a590de57e3eb0357aba4f9468bdc25367"
Sep 05 19:59:55.473044 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:55.473025    2583 scope.go:117] "RemoveContainer" containerID="96b20ef6b8d4235aa5cbabe6129f5a88a821e6dd1996500c04ae70117af7a0f3"
Sep 05 19:59:55.484946 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:55.484918    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"]
Sep 05 19:59:55.485375 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:55.485353    2583 scope.go:117] "RemoveContainer" containerID="2deb0fa2238b4bc1d57ca351206a9b6a590de57e3eb0357aba4f9468bdc25367"
Sep 05 19:59:55.486385 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:59:55.486358    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2deb0fa2238b4bc1d57ca351206a9b6a590de57e3eb0357aba4f9468bdc25367\": container with ID starting with 2deb0fa2238b4bc1d57ca351206a9b6a590de57e3eb0357aba4f9468bdc25367 not found: ID does not exist" containerID="2deb0fa2238b4bc1d57ca351206a9b6a590de57e3eb0357aba4f9468bdc25367"
Sep 05 19:59:55.486514 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:55.486397    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2deb0fa2238b4bc1d57ca351206a9b6a590de57e3eb0357aba4f9468bdc25367"} err="failed to get container status \"2deb0fa2238b4bc1d57ca351206a9b6a590de57e3eb0357aba4f9468bdc25367\": rpc error: code = NotFound desc = could not find container \"2deb0fa2238b4bc1d57ca351206a9b6a590de57e3eb0357aba4f9468bdc25367\": container with ID starting with 2deb0fa2238b4bc1d57ca351206a9b6a590de57e3eb0357aba4f9468bdc25367 not found: ID does not exist"
Sep 05 19:59:55.486514 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:55.486421    2583 scope.go:117] "RemoveContainer" containerID="96b20ef6b8d4235aa5cbabe6129f5a88a821e6dd1996500c04ae70117af7a0f3"
Sep 05 19:59:55.486767 ip-10-0-142-208 kubenswrapper[2583]: E0905 19:59:55.486747    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b20ef6b8d4235aa5cbabe6129f5a88a821e6dd1996500c04ae70117af7a0f3\": container with ID starting with 96b20ef6b8d4235aa5cbabe6129f5a88a821e6dd1996500c04ae70117af7a0f3 not found: ID does not exist" containerID="96b20ef6b8d4235aa5cbabe6129f5a88a821e6dd1996500c04ae70117af7a0f3"
Sep 05 19:59:55.486822 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:55.486778    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b20ef6b8d4235aa5cbabe6129f5a88a821e6dd1996500c04ae70117af7a0f3"} err="failed to get container status \"96b20ef6b8d4235aa5cbabe6129f5a88a821e6dd1996500c04ae70117af7a0f3\": rpc error: code = NotFound desc = could not find container \"96b20ef6b8d4235aa5cbabe6129f5a88a821e6dd1996500c04ae70117af7a0f3\": container with ID starting with 96b20ef6b8d4235aa5cbabe6129f5a88a821e6dd1996500c04ae70117af7a0f3 not found: ID does not exist"
Sep 05 19:59:55.488966 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:55.488908    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b4496e2f-openshift-default-6f7f575cb4-v7jj6"]
Sep 05 19:59:56.656526 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:56.656497    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr"]
Sep 05 19:59:56.656913 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:56.656901    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39a29887-ac3c-4ac9-9760-79e5bc05b9e0" containerName="revoke-keys"
Sep 05 19:59:56.656982 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:56.656914    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="39a29887-ac3c-4ac9-9760-79e5bc05b9e0" containerName="revoke-keys"
Sep 05 19:59:56.656982 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:56.656935    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5af03729-2fae-47c6-a997-87d5d867e133" containerName="istio-proxy"
Sep 05 19:59:56.656982 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:56.656940    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5af03729-2fae-47c6-a997-87d5d867e133" containerName="istio-proxy"
Sep 05 19:59:56.656982 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:56.656947    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e340f762-a090-4588-80bc-4cdf039d1c0b" containerName="payload-pre-processing"
Sep 05 19:59:56.656982 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:56.656953    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e340f762-a090-4588-80bc-4cdf039d1c0b" containerName="payload-pre-processing"
Sep 05 19:59:56.656982 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:56.656970    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5af03729-2fae-47c6-a997-87d5d867e133" containerName="istio-proxy"
Sep 05 19:59:56.656982 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:56.656975    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5af03729-2fae-47c6-a997-87d5d867e133" containerName="istio-proxy"
Sep 05 19:59:56.657252 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:56.657029    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5af03729-2fae-47c6-a997-87d5d867e133" containerName="istio-proxy"
Sep 05 19:59:56.657252 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:56.657038    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="39a29887-ac3c-4ac9-9760-79e5bc05b9e0" containerName="revoke-keys"
Sep 05 19:59:56.657252 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:56.657045    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e340f762-a090-4588-80bc-4cdf039d1c0b" containerName="payload-pre-processing"
Sep 05 19:59:56.661401 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:56.661383    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr"
Sep 05 19:59:56.663480 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:56.663462    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-e1fbf0a6-dockercfg-b4kq9\""
Sep 05 19:59:56.663826 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:56.663807    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-e1fbf0a6\""
Sep 05 19:59:56.671797 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:56.671774    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr"]
Sep 05 19:59:56.743722 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:56.743685    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58607697-0987-4d08-8eb1-b723358ca87b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr\" (UID: \"58607697-0987-4d08-8eb1-b723358ca87b\") " pod="openshift-ingress/payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr"
Sep 05 19:59:56.746242 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:56.746187    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msfqh\" (UniqueName: \"kubernetes.io/projected/58607697-0987-4d08-8eb1-b723358ca87b-kube-api-access-msfqh\") pod \"payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr\" (UID: \"58607697-0987-4d08-8eb1-b723358ca87b\") " pod="openshift-ingress/payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr"
Sep 05 19:59:56.847204 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:56.847168    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58607697-0987-4d08-8eb1-b723358ca87b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr\" (UID: \"58607697-0987-4d08-8eb1-b723358ca87b\") " pod="openshift-ingress/payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr"
Sep 05 19:59:56.847204 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:56.847210    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msfqh\" (UniqueName: \"kubernetes.io/projected/58607697-0987-4d08-8eb1-b723358ca87b-kube-api-access-msfqh\") pod \"payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr\" (UID: \"58607697-0987-4d08-8eb1-b723358ca87b\") " pod="openshift-ingress/payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr"
Sep 05 19:59:56.847875 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:56.847855    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58607697-0987-4d08-8eb1-b723358ca87b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr\" (UID: \"58607697-0987-4d08-8eb1-b723358ca87b\") " pod="openshift-ingress/payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr"
Sep 05 19:59:56.856056 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:56.856025    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msfqh\" (UniqueName: \"kubernetes.io/projected/58607697-0987-4d08-8eb1-b723358ca87b-kube-api-access-msfqh\") pod \"payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr\" (UID: \"58607697-0987-4d08-8eb1-b723358ca87b\") " pod="openshift-ingress/payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr"
Sep 05 19:59:56.974208 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:56.974100    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr"
Sep 05 19:59:57.246889 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:57.246860    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr"]
Sep 05 19:59:57.248487 ip-10-0-142-208 kubenswrapper[2583]: W0905 19:59:57.248454    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58607697_0987_4d08_8eb1_b723358ca87b.slice/crio-7aec0570be101dc0363ceb98be6cd6afcdd790876f71bf54717165aec9cc431a WatchSource:0}: Error finding container 7aec0570be101dc0363ceb98be6cd6afcdd790876f71bf54717165aec9cc431a: Status 404 returned error can't find the container with id 7aec0570be101dc0363ceb98be6cd6afcdd790876f71bf54717165aec9cc431a
Sep 05 19:59:57.436417 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:57.436375    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5af03729-2fae-47c6-a997-87d5d867e133" path="/var/lib/kubelet/pods/5af03729-2fae-47c6-a997-87d5d867e133/volumes"
Sep 05 19:59:57.479953 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:57.479918    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr" event={"ID":"58607697-0987-4d08-8eb1-b723358ca87b","Type":"ContainerStarted","Data":"fe181b0119077aee66cd8b7b1fb942ec83d99ab0ea496ebbe6cd9c6cfd38a434"}
Sep 05 19:59:57.480149 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:57.479971    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr" event={"ID":"58607697-0987-4d08-8eb1-b723358ca87b","Type":"ContainerStarted","Data":"7aec0570be101dc0363ceb98be6cd6afcdd790876f71bf54717165aec9cc431a"}
Sep 05 19:59:57.480149 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:57.480008    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr"
Sep 05 19:59:57.498917 ip-10-0-142-208 kubenswrapper[2583]: I0905 19:59:57.498823    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr" podStartSLOduration=1.498804185 podStartE2EDuration="1.498804185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 19:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 19:59:57.495286 +0000 UTC m=+996.524917907" watchObservedRunningTime="2026-09-05 19:59:57.498804185 +0000 UTC m=+996.528436094"
Sep 05 20:00:00.136872 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:00.136837    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1fd18b-29810640-zv4zk"]
Sep 05 20:00:00.137747 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:00.137723    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5af03729-2fae-47c6-a997-87d5d867e133" containerName="istio-proxy"
Sep 05 20:00:00.144341 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:00.144314    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-588081-29810640-5fvzr"]
Sep 05 20:00:00.144472 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:00.144450    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1fd18b-29810640-zv4zk"
Sep 05 20:00:00.146804 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:00.146782    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zx89\""
Sep 05 20:00:00.148975 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:00.148933    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29810640-dgxzx"]
Sep 05 20:00:00.149084 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:00.149060    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-588081-29810640-5fvzr"
Sep 05 20:00:00.154701 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:00.154679    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1fd18b-29810640-zv4zk"]
Sep 05 20:00:00.154819 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:00.154720    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-588081-29810640-5fvzr"]
Sep 05 20:00:00.154819 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:00.154806    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29810640-dgxzx"
Sep 05 20:00:00.157593 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:00.157503    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29810640-dgxzx"]
Sep 05 20:00:00.160050 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:00.159863    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1fd18b-29810640-zv4zk"
Sep 05 20:00:00.171737 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:00.171717    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-588081-29810640-5fvzr"
Sep 05 20:00:00.223042 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:00.222831    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29810640-dgxzx"
Sep 05 20:00:00.480453 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:00.480358    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1fd18b-29810640-zv4zk"]
Sep 05 20:00:00.484925 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:00:00.484899    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8270f44_b821_4b4a_afab_0638b8b5c210.slice/crio-93d9eef2b9ff1e87c2d0deb88c4d0fe1f8d333fc9603c2e3892605033a08a747 WatchSource:0}: Error finding container 93d9eef2b9ff1e87c2d0deb88c4d0fe1f8d333fc9603c2e3892605033a08a747: Status 404 returned error can't find the container with id 93d9eef2b9ff1e87c2d0deb88c4d0fe1f8d333fc9603c2e3892605033a08a747
Sep 05 20:00:00.501300 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:00.501267    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1fd18b-29810640-zv4zk" event={"ID":"c8270f44-b821-4b4a-afab-0638b8b5c210","Type":"ContainerStarted","Data":"93d9eef2b9ff1e87c2d0deb88c4d0fe1f8d333fc9603c2e3892605033a08a747"}
Sep 05 20:00:00.522593 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:00.522424    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-588081-29810640-5fvzr"]
Sep 05 20:00:00.523891 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:00:00.523861    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod597eb445_16dd_408d_8e84_1efc06db936c.slice/crio-b53b3f07d7884ec34a613dce11818aed65100f90ec1023708dfdbab20be23cc0 WatchSource:0}: Error finding container b53b3f07d7884ec34a613dce11818aed65100f90ec1023708dfdbab20be23cc0: Status 404 returned error can't find the container with id b53b3f07d7884ec34a613dce11818aed65100f90ec1023708dfdbab20be23cc0
Sep 05 20:00:00.555227 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:00.555204    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29810640-dgxzx"]
Sep 05 20:00:00.559293 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:00:00.559271    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5265ef_6e12_4e12_9df5_1f9ab3604f79.slice/crio-7a688b24850eb0bbb9a2e0403738d11e079716d2ee65776f7a2f12b66e5667d3 WatchSource:0}: Error finding container 7a688b24850eb0bbb9a2e0403738d11e079716d2ee65776f7a2f12b66e5667d3: Status 404 returned error can't find the container with id 7a688b24850eb0bbb9a2e0403738d11e079716d2ee65776f7a2f12b66e5667d3
Sep 05 20:00:01.509604 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:01.509494    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1fd18b-29810640-zv4zk" event={"ID":"c8270f44-b821-4b4a-afab-0638b8b5c210","Type":"ContainerStarted","Data":"14f7e293a49ab60c8e0f3d445885f7405fc03439465da1cc26161c30eba00c5c"}
Sep 05 20:00:01.511302 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:01.511272    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29810640-dgxzx" event={"ID":"bb5265ef-6e12-4e12-9df5-1f9ab3604f79","Type":"ContainerStarted","Data":"592943af44d5ddcdec46a02e3cd809b5b5fee6580620ba6534be136ad1156802"}
Sep 05 20:00:01.511439 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:01.511321    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29810640-dgxzx" event={"ID":"bb5265ef-6e12-4e12-9df5-1f9ab3604f79","Type":"ContainerStarted","Data":"7a688b24850eb0bbb9a2e0403738d11e079716d2ee65776f7a2f12b66e5667d3"}
Sep 05 20:00:01.513008 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:01.512979    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-588081-29810640-5fvzr" event={"ID":"597eb445-16dd-408d-8e84-1efc06db936c","Type":"ContainerStarted","Data":"38fdf5855d68371a7c2f298be6aaa2e6c5d7a9a07e9d39e6e1697022308b925e"}
Sep 05 20:00:01.513145 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:01.513037    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-588081-29810640-5fvzr" event={"ID":"597eb445-16dd-408d-8e84-1efc06db936c","Type":"ContainerStarted","Data":"b53b3f07d7884ec34a613dce11818aed65100f90ec1023708dfdbab20be23cc0"}
Sep 05 20:00:01.525408 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:01.525349    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1fd18b-29810640-zv4zk" podStartSLOduration=1.5253321039999999 podStartE2EDuration="1.525332104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:00:01.524680623 +0000 UTC m=+1000.554312519" watchObservedRunningTime="2026-09-05 20:00:01.525332104 +0000 UTC m=+1000.554964011"
Sep 05 20:00:01.541216 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:01.541159    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-588081-29810640-5fvzr" podStartSLOduration=1.541141458 podStartE2EDuration="1.541141458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:00:01.539428591 +0000 UTC m=+1000.569060496" watchObservedRunningTime="2026-09-05 20:00:01.541141458 +0000 UTC m=+1000.570773365"
Sep 05 20:00:01.555515 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:01.555458    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29810640-dgxzx" podStartSLOduration=1.5554425790000002 podStartE2EDuration="1.555442579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:00:01.55341178 +0000 UTC m=+1000.583043685" watchObservedRunningTime="2026-09-05 20:00:01.555442579 +0000 UTC m=+1000.585074521"
Sep 05 20:00:02.520641 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:02.520603    2583 generic.go:356] "Generic (PLEG): container finished" podID="c8270f44-b821-4b4a-afab-0638b8b5c210" containerID="14f7e293a49ab60c8e0f3d445885f7405fc03439465da1cc26161c30eba00c5c" exitCode=0
Sep 05 20:00:02.521098 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:02.520687    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1fd18b-29810640-zv4zk" event={"ID":"c8270f44-b821-4b4a-afab-0638b8b5c210","Type":"ContainerDied","Data":"14f7e293a49ab60c8e0f3d445885f7405fc03439465da1cc26161c30eba00c5c"}
Sep 05 20:00:02.522378 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:02.522355    2583 generic.go:356] "Generic (PLEG): container finished" podID="bb5265ef-6e12-4e12-9df5-1f9ab3604f79" containerID="592943af44d5ddcdec46a02e3cd809b5b5fee6580620ba6534be136ad1156802" exitCode=0
Sep 05 20:00:02.522477 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:02.522406    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29810640-dgxzx" event={"ID":"bb5265ef-6e12-4e12-9df5-1f9ab3604f79","Type":"ContainerDied","Data":"592943af44d5ddcdec46a02e3cd809b5b5fee6580620ba6534be136ad1156802"}
Sep 05 20:00:02.523932 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:02.523907    2583 generic.go:356] "Generic (PLEG): container finished" podID="597eb445-16dd-408d-8e84-1efc06db936c" containerID="38fdf5855d68371a7c2f298be6aaa2e6c5d7a9a07e9d39e6e1697022308b925e" exitCode=0
Sep 05 20:00:02.524022 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:02.523946    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-588081-29810640-5fvzr" event={"ID":"597eb445-16dd-408d-8e84-1efc06db936c","Type":"ContainerDied","Data":"38fdf5855d68371a7c2f298be6aaa2e6c5d7a9a07e9d39e6e1697022308b925e"}
Sep 05 20:00:03.691950 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:03.691915    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29810640-dgxzx"
Sep 05 20:00:03.746452 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:03.746428    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1fd18b-29810640-zv4zk"
Sep 05 20:00:03.749498 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:03.749477    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-588081-29810640-5fvzr"
Sep 05 20:00:04.537321 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:04.537295    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29810640-dgxzx"
Sep 05 20:00:04.537321 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:04.537306    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29810640-dgxzx" event={"ID":"bb5265ef-6e12-4e12-9df5-1f9ab3604f79","Type":"ContainerDied","Data":"7a688b24850eb0bbb9a2e0403738d11e079716d2ee65776f7a2f12b66e5667d3"}
Sep 05 20:00:04.537566 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:04.537363    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a688b24850eb0bbb9a2e0403738d11e079716d2ee65776f7a2f12b66e5667d3"
Sep 05 20:00:04.539009 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:04.538987    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-588081-29810640-5fvzr"
Sep 05 20:00:04.539155 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:04.539015    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-588081-29810640-5fvzr" event={"ID":"597eb445-16dd-408d-8e84-1efc06db936c","Type":"ContainerDied","Data":"b53b3f07d7884ec34a613dce11818aed65100f90ec1023708dfdbab20be23cc0"}
Sep 05 20:00:04.539155 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:04.539060    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b53b3f07d7884ec34a613dce11818aed65100f90ec1023708dfdbab20be23cc0"
Sep 05 20:00:04.540817 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:04.540799    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1fd18b-29810640-zv4zk"
Sep 05 20:00:04.540927 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:04.540846    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1fd18b-29810640-zv4zk" event={"ID":"c8270f44-b821-4b4a-afab-0638b8b5c210","Type":"ContainerDied","Data":"93d9eef2b9ff1e87c2d0deb88c4d0fe1f8d333fc9603c2e3892605033a08a747"}
Sep 05 20:00:04.540927 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:04.540885    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93d9eef2b9ff1e87c2d0deb88c4d0fe1f8d333fc9603c2e3892605033a08a747"
Sep 05 20:00:08.489306 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:08.489273    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr"
Sep 05 20:00:14.028424 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.028376    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"]
Sep 05 20:00:14.028821 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.028807    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb5265ef-6e12-4e12-9df5-1f9ab3604f79" containerName="cleanup"
Sep 05 20:00:14.028870 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.028822    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb5265ef-6e12-4e12-9df5-1f9ab3604f79" containerName="cleanup"
Sep 05 20:00:14.028870 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.028838    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="597eb445-16dd-408d-8e84-1efc06db936c" containerName="cleanup"
Sep 05 20:00:14.028870 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.028845    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="597eb445-16dd-408d-8e84-1efc06db936c" containerName="cleanup"
Sep 05 20:00:14.028870 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.028851    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8270f44-b821-4b4a-afab-0638b8b5c210" containerName="cleanup"
Sep 05 20:00:14.028870 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.028856    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8270f44-b821-4b4a-afab-0638b8b5c210" containerName="cleanup"
Sep 05 20:00:14.029023 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.028916    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="597eb445-16dd-408d-8e84-1efc06db936c" containerName="cleanup"
Sep 05 20:00:14.029023 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.028926    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8270f44-b821-4b4a-afab-0638b8b5c210" containerName="cleanup"
Sep 05 20:00:14.029023 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.028933    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb5265ef-6e12-4e12-9df5-1f9ab3604f79" containerName="cleanup"
Sep 05 20:00:14.034603 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.034582    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:00:14.037066 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.037045    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-c038f394-kserve-self-signed-certs\""
Sep 05 20:00:14.042888 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.042867    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"]
Sep 05 20:00:14.117316 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.117287    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b57496d8-319a-4506-9a23-554147cb8329-home\") pod \"e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq\" (UID: \"b57496d8-319a-4506-9a23-554147cb8329\") " pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:00:14.117457 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.117321    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b57496d8-319a-4506-9a23-554147cb8329-tmp-dir\") pod \"e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq\" (UID: \"b57496d8-319a-4506-9a23-554147cb8329\") " pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:00:14.117457 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.117350    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b57496d8-319a-4506-9a23-554147cb8329-model-cache\") pod \"e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq\" (UID: \"b57496d8-319a-4506-9a23-554147cb8329\") " pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:00:14.117457 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.117394    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b57496d8-319a-4506-9a23-554147cb8329-tls-certs\") pod \"e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq\" (UID: \"b57496d8-319a-4506-9a23-554147cb8329\") " pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:00:14.117457 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.117422    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rkjt\" (UniqueName: \"kubernetes.io/projected/b57496d8-319a-4506-9a23-554147cb8329-kube-api-access-2rkjt\") pod \"e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq\" (UID: \"b57496d8-319a-4506-9a23-554147cb8329\") " pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:00:14.117613 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.117570    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b57496d8-319a-4506-9a23-554147cb8329-dshm\") pod \"e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq\" (UID: \"b57496d8-319a-4506-9a23-554147cb8329\") " pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:00:14.218195 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.218155    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b57496d8-319a-4506-9a23-554147cb8329-dshm\") pod \"e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq\" (UID: \"b57496d8-319a-4506-9a23-554147cb8329\") " pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:00:14.218336 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.218227    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b57496d8-319a-4506-9a23-554147cb8329-home\") pod \"e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq\" (UID: \"b57496d8-319a-4506-9a23-554147cb8329\") " pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:00:14.218336 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.218252    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b57496d8-319a-4506-9a23-554147cb8329-tmp-dir\") pod \"e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq\" (UID: \"b57496d8-319a-4506-9a23-554147cb8329\") " pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:00:14.218336 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.218278    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b57496d8-319a-4506-9a23-554147cb8329-model-cache\") pod \"e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq\" (UID: \"b57496d8-319a-4506-9a23-554147cb8329\") " pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:00:14.218336 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.218318    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b57496d8-319a-4506-9a23-554147cb8329-tls-certs\") pod \"e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq\" (UID: \"b57496d8-319a-4506-9a23-554147cb8329\") " pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:00:14.218543 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.218352    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rkjt\" (UniqueName: \"kubernetes.io/projected/b57496d8-319a-4506-9a23-554147cb8329-kube-api-access-2rkjt\") pod \"e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq\" (UID: \"b57496d8-319a-4506-9a23-554147cb8329\") " pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:00:14.218718 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.218689    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b57496d8-319a-4506-9a23-554147cb8329-home\") pod \"e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq\" (UID: \"b57496d8-319a-4506-9a23-554147cb8329\") " pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:00:14.218848 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.218733    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b57496d8-319a-4506-9a23-554147cb8329-model-cache\") pod \"e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq\" (UID: \"b57496d8-319a-4506-9a23-554147cb8329\") " pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:00:14.218848 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.218761    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b57496d8-319a-4506-9a23-554147cb8329-tmp-dir\") pod \"e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq\" (UID: \"b57496d8-319a-4506-9a23-554147cb8329\") " pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:00:14.220456 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.220434    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b57496d8-319a-4506-9a23-554147cb8329-dshm\") pod \"e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq\" (UID: \"b57496d8-319a-4506-9a23-554147cb8329\") " pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:00:14.220747 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.220727    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b57496d8-319a-4506-9a23-554147cb8329-tls-certs\") pod \"e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq\" (UID: \"b57496d8-319a-4506-9a23-554147cb8329\") " pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:00:14.237808 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.237779    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rkjt\" (UniqueName: \"kubernetes.io/projected/b57496d8-319a-4506-9a23-554147cb8329-kube-api-access-2rkjt\") pod \"e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq\" (UID: \"b57496d8-319a-4506-9a23-554147cb8329\") " pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:00:14.344400 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.344359    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:00:14.624511 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:14.624480    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"]
Sep 05 20:00:14.625505 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:00:14.625475    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb57496d8_319a_4506_9a23_554147cb8329.slice/crio-5339d18bcd786a2ec505763af35008634a1ed28b9037d7c6cbdebaccef25efd4 WatchSource:0}: Error finding container 5339d18bcd786a2ec505763af35008634a1ed28b9037d7c6cbdebaccef25efd4: Status 404 returned error can't find the container with id 5339d18bcd786a2ec505763af35008634a1ed28b9037d7c6cbdebaccef25efd4
Sep 05 20:00:15.612638 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:15.612598    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq" event={"ID":"b57496d8-319a-4506-9a23-554147cb8329","Type":"ContainerStarted","Data":"df4e5551c2d4528ec0f96b4223aa23c113a1b489946328cfdebde89e29d64b8e"}
Sep 05 20:00:15.613002 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:15.612648    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq" event={"ID":"b57496d8-319a-4506-9a23-554147cb8329","Type":"ContainerStarted","Data":"5339d18bcd786a2ec505763af35008634a1ed28b9037d7c6cbdebaccef25efd4"}
Sep 05 20:00:15.631682 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:15.631632    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq" podStartSLOduration=1.631615681 podStartE2EDuration="1.631615681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:00:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:00:15.628790149 +0000 UTC m=+1014.658422054" watchObservedRunningTime="2026-09-05 20:00:15.631615681 +0000 UTC m=+1014.661247587"
Sep 05 20:00:20.807175 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:20.807138    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1fbf0a6-md7xj"]
Sep 05 20:00:20.811042 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:20.811018    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1fbf0a6-md7xj"
Sep 05 20:00:20.813218 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:20.813194    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 20:00:20.813218 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:20.813205    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zx89\""
Sep 05 20:00:20.821743 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:20.821700    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1fbf0a6-md7xj"]
Sep 05 20:00:20.883022 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:20.882972    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe6eccb-e32e-4801-8d58-eb0d49d29f1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e1fbf0a6-md7xj\" (UID: \"bfe6eccb-e32e-4801-8d58-eb0d49d29f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1fbf0a6-md7xj"
Sep 05 20:00:20.984343 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:20.984303    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe6eccb-e32e-4801-8d58-eb0d49d29f1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e1fbf0a6-md7xj\" (UID: \"bfe6eccb-e32e-4801-8d58-eb0d49d29f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1fbf0a6-md7xj"
Sep 05 20:00:20.985280 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:20.985249    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe6eccb-e32e-4801-8d58-eb0d49d29f1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e1fbf0a6-md7xj\" (UID: \"bfe6eccb-e32e-4801-8d58-eb0d49d29f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1fbf0a6-md7xj"
Sep 05 20:00:21.125888 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:21.125854    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1fbf0a6-md7xj"
Sep 05 20:00:21.399164 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:21.399138    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1fbf0a6-md7xj"]
Sep 05 20:00:21.400608 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:00:21.400583    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe6eccb_e32e_4801_8d58_eb0d49d29f1d.slice/crio-815fec5431516a2600b86db92e78120e6c67f14b7e60530219e07d5a777bdb04 WatchSource:0}: Error finding container 815fec5431516a2600b86db92e78120e6c67f14b7e60530219e07d5a777bdb04: Status 404 returned error can't find the container with id 815fec5431516a2600b86db92e78120e6c67f14b7e60530219e07d5a777bdb04
Sep 05 20:00:21.653361 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:21.653250    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1fbf0a6-md7xj" event={"ID":"bfe6eccb-e32e-4801-8d58-eb0d49d29f1d","Type":"ContainerStarted","Data":"815fec5431516a2600b86db92e78120e6c67f14b7e60530219e07d5a777bdb04"}
Sep 05 20:00:22.661167 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:22.661133    2583 generic.go:356] "Generic (PLEG): container finished" podID="bfe6eccb-e32e-4801-8d58-eb0d49d29f1d" containerID="585974f0690c4bc7badb29e2afdf5b69dc114d5c1ce9c125f610c833d63b6667" exitCode=0
Sep 05 20:00:22.661523 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:22.661195    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1fbf0a6-md7xj" event={"ID":"bfe6eccb-e32e-4801-8d58-eb0d49d29f1d","Type":"ContainerDied","Data":"585974f0690c4bc7badb29e2afdf5b69dc114d5c1ce9c125f610c833d63b6667"}
Sep 05 20:00:23.815662 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:23.815636    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1fbf0a6-md7xj"
Sep 05 20:00:23.912424 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:23.912391    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe6eccb-e32e-4801-8d58-eb0d49d29f1d-maas-api-service-ca\") pod \"bfe6eccb-e32e-4801-8d58-eb0d49d29f1d\" (UID: \"bfe6eccb-e32e-4801-8d58-eb0d49d29f1d\") "
Sep 05 20:00:23.912763 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:23.912738    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe6eccb-e32e-4801-8d58-eb0d49d29f1d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bfe6eccb-e32e-4801-8d58-eb0d49d29f1d" (UID: "bfe6eccb-e32e-4801-8d58-eb0d49d29f1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:00:24.013254 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:24.013178    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfe6eccb-e32e-4801-8d58-eb0d49d29f1d-maas-api-service-ca\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:00:24.344647 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:24.344619    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:00:24.344887 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:24.344853    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:00:24.357826 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:24.357800    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:00:24.676901 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:24.676812    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1fbf0a6-md7xj"
Sep 05 20:00:24.676901 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:24.676834    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1fbf0a6-md7xj" event={"ID":"bfe6eccb-e32e-4801-8d58-eb0d49d29f1d","Type":"ContainerDied","Data":"815fec5431516a2600b86db92e78120e6c67f14b7e60530219e07d5a777bdb04"}
Sep 05 20:00:24.676901 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:24.676878    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="815fec5431516a2600b86db92e78120e6c67f14b7e60530219e07d5a777bdb04"
Sep 05 20:00:24.691366 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:24.691342    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:00:27.415390 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:27.415257    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr"]
Sep 05 20:00:27.415862 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:27.415564    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr" podUID="58607697-0987-4d08-8eb1-b723358ca87b" containerName="payload-processing" containerID="cri-o://fe181b0119077aee66cd8b7b1fb942ec83d99ab0ea496ebbe6cd9c6cfd38a434" gracePeriod=30
Sep 05 20:00:27.558539 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:27.558485    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1fbf0a6-md7xj"]
Sep 05 20:00:27.561375 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:27.561342    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e1fbf0a6-md7xj"]
Sep 05 20:00:27.685058 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:27.684969    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr"
Sep 05 20:00:27.702858 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:27.702825    2583 generic.go:356] "Generic (PLEG): container finished" podID="58607697-0987-4d08-8eb1-b723358ca87b" containerID="fe181b0119077aee66cd8b7b1fb942ec83d99ab0ea496ebbe6cd9c6cfd38a434" exitCode=0
Sep 05 20:00:27.703093 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:27.702873    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr"
Sep 05 20:00:27.703544 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:27.702891    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr" event={"ID":"58607697-0987-4d08-8eb1-b723358ca87b","Type":"ContainerDied","Data":"fe181b0119077aee66cd8b7b1fb942ec83d99ab0ea496ebbe6cd9c6cfd38a434"}
Sep 05 20:00:27.703544 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:27.703417    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr" event={"ID":"58607697-0987-4d08-8eb1-b723358ca87b","Type":"ContainerDied","Data":"7aec0570be101dc0363ceb98be6cd6afcdd790876f71bf54717165aec9cc431a"}
Sep 05 20:00:27.703544 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:27.703479    2583 scope.go:117] "RemoveContainer" containerID="fe181b0119077aee66cd8b7b1fb942ec83d99ab0ea496ebbe6cd9c6cfd38a434"
Sep 05 20:00:27.715166 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:27.715147    2583 scope.go:117] "RemoveContainer" containerID="fe181b0119077aee66cd8b7b1fb942ec83d99ab0ea496ebbe6cd9c6cfd38a434"
Sep 05 20:00:27.715507 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:00:27.715482    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe181b0119077aee66cd8b7b1fb942ec83d99ab0ea496ebbe6cd9c6cfd38a434\": container with ID starting with fe181b0119077aee66cd8b7b1fb942ec83d99ab0ea496ebbe6cd9c6cfd38a434 not found: ID does not exist" containerID="fe181b0119077aee66cd8b7b1fb942ec83d99ab0ea496ebbe6cd9c6cfd38a434"
Sep 05 20:00:27.715600 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:27.715511    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe181b0119077aee66cd8b7b1fb942ec83d99ab0ea496ebbe6cd9c6cfd38a434"} err="failed to get container status \"fe181b0119077aee66cd8b7b1fb942ec83d99ab0ea496ebbe6cd9c6cfd38a434\": rpc error: code = NotFound desc = could not find container \"fe181b0119077aee66cd8b7b1fb942ec83d99ab0ea496ebbe6cd9c6cfd38a434\": container with ID starting with fe181b0119077aee66cd8b7b1fb942ec83d99ab0ea496ebbe6cd9c6cfd38a434 not found: ID does not exist"
Sep 05 20:00:27.750093 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:27.750068    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58607697-0987-4d08-8eb1-b723358ca87b-plugins-config-volume\") pod \"58607697-0987-4d08-8eb1-b723358ca87b\" (UID: \"58607697-0987-4d08-8eb1-b723358ca87b\") "
Sep 05 20:00:27.750249 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:27.750163    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msfqh\" (UniqueName: \"kubernetes.io/projected/58607697-0987-4d08-8eb1-b723358ca87b-kube-api-access-msfqh\") pod \"58607697-0987-4d08-8eb1-b723358ca87b\" (UID: \"58607697-0987-4d08-8eb1-b723358ca87b\") "
Sep 05 20:00:27.750504 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:27.750479    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58607697-0987-4d08-8eb1-b723358ca87b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "58607697-0987-4d08-8eb1-b723358ca87b" (UID: "58607697-0987-4d08-8eb1-b723358ca87b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:00:27.750558 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:27.750546    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58607697-0987-4d08-8eb1-b723358ca87b-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:00:27.752034 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:27.752017    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58607697-0987-4d08-8eb1-b723358ca87b-kube-api-access-msfqh" (OuterVolumeSpecName: "kube-api-access-msfqh") pod "58607697-0987-4d08-8eb1-b723358ca87b" (UID: "58607697-0987-4d08-8eb1-b723358ca87b"). InnerVolumeSpecName "kube-api-access-msfqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:00:27.851285 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:27.851253    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msfqh\" (UniqueName: \"kubernetes.io/projected/58607697-0987-4d08-8eb1-b723358ca87b-kube-api-access-msfqh\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:00:28.025165 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:28.025134    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr"]
Sep 05 20:00:28.030876 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:28.030849    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e1fbf0a6-6fcf5589d-65lzr"]
Sep 05 20:00:29.435216 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:29.435186    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58607697-0987-4d08-8eb1-b723358ca87b" path="/var/lib/kubelet/pods/58607697-0987-4d08-8eb1-b723358ca87b/volumes"
Sep 05 20:00:29.435578 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:29.435565    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfe6eccb-e32e-4801-8d58-eb0d49d29f1d" path="/var/lib/kubelet/pods/bfe6eccb-e32e-4801-8d58-eb0d49d29f1d/volumes"
Sep 05 20:00:32.844446 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:32.844407    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn"]
Sep 05 20:00:32.844898 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:32.844844    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58607697-0987-4d08-8eb1-b723358ca87b" containerName="payload-processing"
Sep 05 20:00:32.844898 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:32.844856    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="58607697-0987-4d08-8eb1-b723358ca87b" containerName="payload-processing"
Sep 05 20:00:32.844898 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:32.844869    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfe6eccb-e32e-4801-8d58-eb0d49d29f1d" containerName="revoke-keys"
Sep 05 20:00:32.844898 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:32.844875    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfe6eccb-e32e-4801-8d58-eb0d49d29f1d" containerName="revoke-keys"
Sep 05 20:00:32.845041 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:32.844951    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfe6eccb-e32e-4801-8d58-eb0d49d29f1d" containerName="revoke-keys"
Sep 05 20:00:32.845041 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:32.844961    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="58607697-0987-4d08-8eb1-b723358ca87b" containerName="payload-processing"
Sep 05 20:00:32.849699 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:32.849678    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn"
Sep 05 20:00:32.852277 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:32.852257    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a7428d13\""
Sep 05 20:00:32.862093 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:32.862063    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn"]
Sep 05 20:00:32.906586 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:32.906552    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75gzw\" (UniqueName: \"kubernetes.io/projected/7d71d311-2ee9-41b3-a646-222c5eb5702b-kube-api-access-75gzw\") pod \"payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn\" (UID: \"7d71d311-2ee9-41b3-a646-222c5eb5702b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn"
Sep 05 20:00:32.906758 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:32.906637    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d71d311-2ee9-41b3-a646-222c5eb5702b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn\" (UID: \"7d71d311-2ee9-41b3-a646-222c5eb5702b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn"
Sep 05 20:00:33.007337 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:33.007308    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75gzw\" (UniqueName: \"kubernetes.io/projected/7d71d311-2ee9-41b3-a646-222c5eb5702b-kube-api-access-75gzw\") pod \"payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn\" (UID: \"7d71d311-2ee9-41b3-a646-222c5eb5702b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn"
Sep 05 20:00:33.007511 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:33.007381    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d71d311-2ee9-41b3-a646-222c5eb5702b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn\" (UID: \"7d71d311-2ee9-41b3-a646-222c5eb5702b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn"
Sep 05 20:00:33.008377 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:33.008350    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d71d311-2ee9-41b3-a646-222c5eb5702b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn\" (UID: \"7d71d311-2ee9-41b3-a646-222c5eb5702b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn"
Sep 05 20:00:33.015769 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:33.015739    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75gzw\" (UniqueName: \"kubernetes.io/projected/7d71d311-2ee9-41b3-a646-222c5eb5702b-kube-api-access-75gzw\") pod \"payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn\" (UID: \"7d71d311-2ee9-41b3-a646-222c5eb5702b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn"
Sep 05 20:00:33.161163 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:33.161052    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn"
Sep 05 20:00:33.435055 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:00:33.435024    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d71d311_2ee9_41b3_a646_222c5eb5702b.slice/crio-f7a23a61ba710b11947a785b864260d0e6969244fa02843166ce5171f1a727e7 WatchSource:0}: Error finding container f7a23a61ba710b11947a785b864260d0e6969244fa02843166ce5171f1a727e7: Status 404 returned error can't find the container with id f7a23a61ba710b11947a785b864260d0e6969244fa02843166ce5171f1a727e7
Sep 05 20:00:33.436239 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:33.436213    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn"]
Sep 05 20:00:33.744396 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:33.744294    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn" event={"ID":"7d71d311-2ee9-41b3-a646-222c5eb5702b","Type":"ContainerStarted","Data":"928eee2593ecc203d9ba1ff973c12217a9f23625400ecbabc249b1ce1ff71655"}
Sep 05 20:00:33.744396 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:33.744349    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn" event={"ID":"7d71d311-2ee9-41b3-a646-222c5eb5702b","Type":"ContainerStarted","Data":"f7a23a61ba710b11947a785b864260d0e6969244fa02843166ce5171f1a727e7"}
Sep 05 20:00:33.744396 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:33.744371    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn"
Sep 05 20:00:33.763438 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:33.763391    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn" podStartSLOduration=1.763376295 podStartE2EDuration="1.763376295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:00:33.761028877 +0000 UTC m=+1032.790660783" watchObservedRunningTime="2026-09-05 20:00:33.763376295 +0000 UTC m=+1032.793008200"
Sep 05 20:00:44.753304 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:44.753267    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn"
Sep 05 20:00:56.644420 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:56.644375    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7428d13-svpx7"]
Sep 05 20:00:56.654901 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:56.654860    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7428d13-svpx7"
Sep 05 20:00:56.655924 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:56.655894    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7428d13-svpx7"]
Sep 05 20:00:56.657126 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:56.657056    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zx89\""
Sep 05 20:00:56.657283 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:56.657056    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 20:00:56.754621 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:56.754584    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeeccd7a-dc53-4121-ba2b-7cd4407fde61-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a7428d13-svpx7\" (UID: \"eeeccd7a-dc53-4121-ba2b-7cd4407fde61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7428d13-svpx7"
Sep 05 20:00:56.855315 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:56.855270    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeeccd7a-dc53-4121-ba2b-7cd4407fde61-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a7428d13-svpx7\" (UID: \"eeeccd7a-dc53-4121-ba2b-7cd4407fde61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7428d13-svpx7"
Sep 05 20:00:56.855908 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:56.855885    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeeccd7a-dc53-4121-ba2b-7cd4407fde61-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a7428d13-svpx7\" (UID: \"eeeccd7a-dc53-4121-ba2b-7cd4407fde61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7428d13-svpx7"
Sep 05 20:00:56.968662 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:56.968568    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7428d13-svpx7"
Sep 05 20:00:57.241021 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:57.240996    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7428d13-svpx7"]
Sep 05 20:00:57.242945 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:00:57.242921    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeeccd7a_dc53_4121_ba2b_7cd4407fde61.slice/crio-c1128756df7d504826b9258b19c337d70f8cdf6f5c5a2e01c748b71cc676dfe8 WatchSource:0}: Error finding container c1128756df7d504826b9258b19c337d70f8cdf6f5c5a2e01c748b71cc676dfe8: Status 404 returned error can't find the container with id c1128756df7d504826b9258b19c337d70f8cdf6f5c5a2e01c748b71cc676dfe8
Sep 05 20:00:57.912593 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:57.912494    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7428d13-svpx7" event={"ID":"eeeccd7a-dc53-4121-ba2b-7cd4407fde61","Type":"ContainerStarted","Data":"714ce1318b52a993c4d1e67ef8db3a8cad51831fc620ada52a133189eaa8282f"}
Sep 05 20:00:57.912593 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:57.912552    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7428d13-svpx7" event={"ID":"eeeccd7a-dc53-4121-ba2b-7cd4407fde61","Type":"ContainerStarted","Data":"c1128756df7d504826b9258b19c337d70f8cdf6f5c5a2e01c748b71cc676dfe8"}
Sep 05 20:00:57.929463 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:57.929407    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7428d13-svpx7" podStartSLOduration=1.9293927929999999 podStartE2EDuration="1.929392793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:00:57.92738661 +0000 UTC m=+1056.957018527" watchObservedRunningTime="2026-09-05 20:00:57.929392793 +0000 UTC m=+1056.959024735"
Sep 05 20:00:58.921788 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:58.921753    2583 generic.go:356] "Generic (PLEG): container finished" podID="eeeccd7a-dc53-4121-ba2b-7cd4407fde61" containerID="714ce1318b52a993c4d1e67ef8db3a8cad51831fc620ada52a133189eaa8282f" exitCode=0
Sep 05 20:00:58.922201 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:00:58.921840    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7428d13-svpx7" event={"ID":"eeeccd7a-dc53-4121-ba2b-7cd4407fde61","Type":"ContainerDied","Data":"714ce1318b52a993c4d1e67ef8db3a8cad51831fc620ada52a133189eaa8282f"}
Sep 05 20:01:00.068039 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:00.068012    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7428d13-svpx7"
Sep 05 20:01:00.189204 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:00.189098    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeeccd7a-dc53-4121-ba2b-7cd4407fde61-maas-api-service-ca\") pod \"eeeccd7a-dc53-4121-ba2b-7cd4407fde61\" (UID: \"eeeccd7a-dc53-4121-ba2b-7cd4407fde61\") "
Sep 05 20:01:00.189461 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:00.189439    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeeccd7a-dc53-4121-ba2b-7cd4407fde61-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eeeccd7a-dc53-4121-ba2b-7cd4407fde61" (UID: "eeeccd7a-dc53-4121-ba2b-7cd4407fde61"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:01:00.289961 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:00.289926    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeeccd7a-dc53-4121-ba2b-7cd4407fde61-maas-api-service-ca\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:01:00.934466 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:00.934433    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7428d13-svpx7" event={"ID":"eeeccd7a-dc53-4121-ba2b-7cd4407fde61","Type":"ContainerDied","Data":"c1128756df7d504826b9258b19c337d70f8cdf6f5c5a2e01c748b71cc676dfe8"}
Sep 05 20:01:00.934633 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:00.934481    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1128756df7d504826b9258b19c337d70f8cdf6f5c5a2e01c748b71cc676dfe8"
Sep 05 20:01:00.934633 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:00.934448    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7428d13-svpx7"
Sep 05 20:01:01.358740 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:01.358703    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1fd18b-wbsz5"]
Sep 05 20:01:01.359419 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:01.359400    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeeccd7a-dc53-4121-ba2b-7cd4407fde61" containerName="revoke-keys"
Sep 05 20:01:01.359489 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:01.359421    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeeccd7a-dc53-4121-ba2b-7cd4407fde61" containerName="revoke-keys"
Sep 05 20:01:01.359541 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:01.359530    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeeccd7a-dc53-4121-ba2b-7cd4407fde61" containerName="revoke-keys"
Sep 05 20:01:01.363000 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:01.362986    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1fd18b-wbsz5"
Sep 05 20:01:01.365720 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:01.365687    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 20:01:01.365984 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:01.365962    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zx89\""
Sep 05 20:01:01.370877 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:01.370851    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1fd18b-wbsz5"]
Sep 05 20:01:01.401418 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:01.401384    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0167081-17d9-4e12-aa99-16e2b0aeb582-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-1fd18b-wbsz5\" (UID: \"b0167081-17d9-4e12-aa99-16e2b0aeb582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1fd18b-wbsz5"
Sep 05 20:01:01.502579 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:01.502546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0167081-17d9-4e12-aa99-16e2b0aeb582-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-1fd18b-wbsz5\" (UID: \"b0167081-17d9-4e12-aa99-16e2b0aeb582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1fd18b-wbsz5"
Sep 05 20:01:01.503200 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:01.503177    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0167081-17d9-4e12-aa99-16e2b0aeb582-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-1fd18b-wbsz5\" (UID: \"b0167081-17d9-4e12-aa99-16e2b0aeb582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1fd18b-wbsz5"
Sep 05 20:01:01.677700 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:01.677612    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1fd18b-wbsz5"
Sep 05 20:01:01.951121 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:01.951076    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1fd18b-wbsz5"]
Sep 05 20:01:01.952195 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:01:01.952165    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0167081_17d9_4e12_aa99_16e2b0aeb582.slice/crio-24faf01cf09e3b8d0f921cb8d31e65cd2acbf0664c1f93575a7119f90ea6ee78 WatchSource:0}: Error finding container 24faf01cf09e3b8d0f921cb8d31e65cd2acbf0664c1f93575a7119f90ea6ee78: Status 404 returned error can't find the container with id 24faf01cf09e3b8d0f921cb8d31e65cd2acbf0664c1f93575a7119f90ea6ee78
Sep 05 20:01:02.751019 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:02.750985    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn"]
Sep 05 20:01:02.751505 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:02.751257    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn" podUID="7d71d311-2ee9-41b3-a646-222c5eb5702b" containerName="payload-pre-processing" containerID="cri-o://928eee2593ecc203d9ba1ff973c12217a9f23625400ecbabc249b1ce1ff71655" gracePeriod=30
Sep 05 20:01:02.907997 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:02.907958    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7428d13-svpx7"]
Sep 05 20:01:02.911537 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:02.911516    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a7428d13-svpx7"]
Sep 05 20:01:02.952449 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:02.952411    2583 generic.go:356] "Generic (PLEG): container finished" podID="7d71d311-2ee9-41b3-a646-222c5eb5702b" containerID="928eee2593ecc203d9ba1ff973c12217a9f23625400ecbabc249b1ce1ff71655" exitCode=0
Sep 05 20:01:02.952632 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:02.952467    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn" event={"ID":"7d71d311-2ee9-41b3-a646-222c5eb5702b","Type":"ContainerDied","Data":"928eee2593ecc203d9ba1ff973c12217a9f23625400ecbabc249b1ce1ff71655"}
Sep 05 20:01:02.955442 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:02.955410    2583 generic.go:356] "Generic (PLEG): container finished" podID="b0167081-17d9-4e12-aa99-16e2b0aeb582" containerID="870e6386332b6e10ef146e4a9b0908d734115d415cc0b64279e347c09807ea7c" exitCode=0
Sep 05 20:01:02.955564 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:02.955453    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1fd18b-wbsz5" event={"ID":"b0167081-17d9-4e12-aa99-16e2b0aeb582","Type":"ContainerDied","Data":"870e6386332b6e10ef146e4a9b0908d734115d415cc0b64279e347c09807ea7c"}
Sep 05 20:01:02.955564 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:02.955498    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1fd18b-wbsz5" event={"ID":"b0167081-17d9-4e12-aa99-16e2b0aeb582","Type":"ContainerStarted","Data":"24faf01cf09e3b8d0f921cb8d31e65cd2acbf0664c1f93575a7119f90ea6ee78"}
Sep 05 20:01:03.021661 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:03.021634    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn"
Sep 05 20:01:03.117095 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:03.117061    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d71d311-2ee9-41b3-a646-222c5eb5702b-plugins-config-volume\") pod \"7d71d311-2ee9-41b3-a646-222c5eb5702b\" (UID: \"7d71d311-2ee9-41b3-a646-222c5eb5702b\") "
Sep 05 20:01:03.117251 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:03.117101    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75gzw\" (UniqueName: \"kubernetes.io/projected/7d71d311-2ee9-41b3-a646-222c5eb5702b-kube-api-access-75gzw\") pod \"7d71d311-2ee9-41b3-a646-222c5eb5702b\" (UID: \"7d71d311-2ee9-41b3-a646-222c5eb5702b\") "
Sep 05 20:01:03.117442 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:03.117408    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d71d311-2ee9-41b3-a646-222c5eb5702b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7d71d311-2ee9-41b3-a646-222c5eb5702b" (UID: "7d71d311-2ee9-41b3-a646-222c5eb5702b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:01:03.119193 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:03.119168    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d71d311-2ee9-41b3-a646-222c5eb5702b-kube-api-access-75gzw" (OuterVolumeSpecName: "kube-api-access-75gzw") pod "7d71d311-2ee9-41b3-a646-222c5eb5702b" (UID: "7d71d311-2ee9-41b3-a646-222c5eb5702b"). InnerVolumeSpecName "kube-api-access-75gzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:01:03.218200 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:03.218165    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d71d311-2ee9-41b3-a646-222c5eb5702b-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:01:03.218200 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:03.218195    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75gzw\" (UniqueName: \"kubernetes.io/projected/7d71d311-2ee9-41b3-a646-222c5eb5702b-kube-api-access-75gzw\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:01:03.435281 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:03.435248    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeeccd7a-dc53-4121-ba2b-7cd4407fde61" path="/var/lib/kubelet/pods/eeeccd7a-dc53-4121-ba2b-7cd4407fde61/volumes"
Sep 05 20:01:03.963903 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:03.963873    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn"
Sep 05 20:01:03.965819 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:03.965784    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn" event={"ID":"7d71d311-2ee9-41b3-a646-222c5eb5702b","Type":"ContainerDied","Data":"f7a23a61ba710b11947a785b864260d0e6969244fa02843166ce5171f1a727e7"}
Sep 05 20:01:03.965956 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:03.965865    2583 scope.go:117] "RemoveContainer" containerID="928eee2593ecc203d9ba1ff973c12217a9f23625400ecbabc249b1ce1ff71655"
Sep 05 20:01:03.990083 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:03.990048    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn"]
Sep 05 20:01:03.995361 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:03.995327    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a7428d13-c87b57746-wnttn"]
Sep 05 20:01:04.130761 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:04.130730    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1fd18b-wbsz5"
Sep 05 20:01:04.227268 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:04.227175    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0167081-17d9-4e12-aa99-16e2b0aeb582-maas-api-service-ca\") pod \"b0167081-17d9-4e12-aa99-16e2b0aeb582\" (UID: \"b0167081-17d9-4e12-aa99-16e2b0aeb582\") "
Sep 05 20:01:04.233946 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:04.227966    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0167081-17d9-4e12-aa99-16e2b0aeb582-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b0167081-17d9-4e12-aa99-16e2b0aeb582" (UID: "b0167081-17d9-4e12-aa99-16e2b0aeb582"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:01:04.328515 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:04.328481    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0167081-17d9-4e12-aa99-16e2b0aeb582-maas-api-service-ca\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:01:04.972895 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:04.972864    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1fd18b-wbsz5"
Sep 05 20:01:04.973299 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:04.972903    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1fd18b-wbsz5" event={"ID":"b0167081-17d9-4e12-aa99-16e2b0aeb582","Type":"ContainerDied","Data":"24faf01cf09e3b8d0f921cb8d31e65cd2acbf0664c1f93575a7119f90ea6ee78"}
Sep 05 20:01:04.973299 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:04.972950    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24faf01cf09e3b8d0f921cb8d31e65cd2acbf0664c1f93575a7119f90ea6ee78"
Sep 05 20:01:05.435938 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:05.435902    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d71d311-2ee9-41b3-a646-222c5eb5702b" path="/var/lib/kubelet/pods/7d71d311-2ee9-41b3-a646-222c5eb5702b/volumes"
Sep 05 20:01:05.708219 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:05.708134    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"]
Sep 05 20:01:08.997527 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:08.997493    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9"]
Sep 05 20:01:08.997917 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:08.997904    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d71d311-2ee9-41b3-a646-222c5eb5702b" containerName="payload-pre-processing"
Sep 05 20:01:08.997960 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:08.997917    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d71d311-2ee9-41b3-a646-222c5eb5702b" containerName="payload-pre-processing"
Sep 05 20:01:08.997960 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:08.997925    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0167081-17d9-4e12-aa99-16e2b0aeb582" containerName="revoke-keys"
Sep 05 20:01:08.997960 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:08.997931    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0167081-17d9-4e12-aa99-16e2b0aeb582" containerName="revoke-keys"
Sep 05 20:01:08.998053 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:08.998014    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0167081-17d9-4e12-aa99-16e2b0aeb582" containerName="revoke-keys"
Sep 05 20:01:08.998053 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:08.998023    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d71d311-2ee9-41b3-a646-222c5eb5702b" containerName="payload-pre-processing"
Sep 05 20:01:09.002471 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:09.002449    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9"
Sep 05 20:01:09.004774 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:09.004755    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-c06406ce\""
Sep 05 20:01:09.004883 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:09.004790    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-c06406ce-dockercfg-2qvwg\""
Sep 05 20:01:09.015577 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:09.015553    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9"]
Sep 05 20:01:09.022005 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:09.021982    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt"]
Sep 05 20:01:09.025469 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:09.025451    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt"
Sep 05 20:01:09.047372 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:09.043637    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt"]
Sep 05 20:01:09.078189 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:09.078157    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgdxw\" (UniqueName: \"kubernetes.io/projected/61ced69f-fa9e-4a79-9a66-dfc7d276b99c-kube-api-access-vgdxw\") pod \"payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt\" (UID: \"61ced69f-fa9e-4a79-9a66-dfc7d276b99c\") " pod="openshift-ingress/payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt"
Sep 05 20:01:09.078360 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:09.078231    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a94483b-1654-4f71-b48d-869d3a0f36e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9\" (UID: \"9a94483b-1654-4f71-b48d-869d3a0f36e4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9"
Sep 05 20:01:09.078360 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:09.078274    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61ced69f-fa9e-4a79-9a66-dfc7d276b99c-plugins-config-volume\") pod \"payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt\" (UID: \"61ced69f-fa9e-4a79-9a66-dfc7d276b99c\") " pod="openshift-ingress/payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt"
Sep 05 20:01:09.078459 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:09.078367    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4vkk\" (UniqueName: \"kubernetes.io/projected/9a94483b-1654-4f71-b48d-869d3a0f36e4-kube-api-access-g4vkk\") pod \"payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9\" (UID: \"9a94483b-1654-4f71-b48d-869d3a0f36e4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9"
Sep 05 20:01:09.179486 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:09.179447    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a94483b-1654-4f71-b48d-869d3a0f36e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9\" (UID: \"9a94483b-1654-4f71-b48d-869d3a0f36e4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9"
Sep 05 20:01:09.179670 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:09.179504    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61ced69f-fa9e-4a79-9a66-dfc7d276b99c-plugins-config-volume\") pod \"payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt\" (UID: \"61ced69f-fa9e-4a79-9a66-dfc7d276b99c\") " pod="openshift-ingress/payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt"
Sep 05 20:01:09.179670 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:09.179560    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4vkk\" (UniqueName: \"kubernetes.io/projected/9a94483b-1654-4f71-b48d-869d3a0f36e4-kube-api-access-g4vkk\") pod \"payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9\" (UID: \"9a94483b-1654-4f71-b48d-869d3a0f36e4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9"
Sep 05 20:01:09.179670 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:09.179630    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgdxw\" (UniqueName: \"kubernetes.io/projected/61ced69f-fa9e-4a79-9a66-dfc7d276b99c-kube-api-access-vgdxw\") pod \"payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt\" (UID: \"61ced69f-fa9e-4a79-9a66-dfc7d276b99c\") " pod="openshift-ingress/payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt"
Sep 05 20:01:09.180299 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:09.180275    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61ced69f-fa9e-4a79-9a66-dfc7d276b99c-plugins-config-volume\") pod \"payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt\" (UID: \"61ced69f-fa9e-4a79-9a66-dfc7d276b99c\") " pod="openshift-ingress/payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt"
Sep 05 20:01:09.180349 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:09.180309    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a94483b-1654-4f71-b48d-869d3a0f36e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9\" (UID: \"9a94483b-1654-4f71-b48d-869d3a0f36e4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9"
Sep 05 20:01:09.187275 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:09.187247    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4vkk\" (UniqueName: \"kubernetes.io/projected/9a94483b-1654-4f71-b48d-869d3a0f36e4-kube-api-access-g4vkk\") pod \"payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9\" (UID: \"9a94483b-1654-4f71-b48d-869d3a0f36e4\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9"
Sep 05 20:01:09.187275 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:09.187261    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgdxw\" (UniqueName: \"kubernetes.io/projected/61ced69f-fa9e-4a79-9a66-dfc7d276b99c-kube-api-access-vgdxw\") pod \"payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt\" (UID: \"61ced69f-fa9e-4a79-9a66-dfc7d276b99c\") " pod="openshift-ingress/payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt"
Sep 05 20:01:09.313639 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:09.313539    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9"
Sep 05 20:01:09.336836 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:09.336799    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt"
Sep 05 20:01:09.609012 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:09.608981    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9"]
Sep 05 20:01:09.611096 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:01:09.611062    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a94483b_1654_4f71_b48d_869d3a0f36e4.slice/crio-98e007bb5b377ec3c0a290baa2b5e335b2d1eab73a8aae62df6350083532af95 WatchSource:0}: Error finding container 98e007bb5b377ec3c0a290baa2b5e335b2d1eab73a8aae62df6350083532af95: Status 404 returned error can't find the container with id 98e007bb5b377ec3c0a290baa2b5e335b2d1eab73a8aae62df6350083532af95
Sep 05 20:01:09.634614 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:01:09.634583    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ced69f_fa9e_4a79_9a66_dfc7d276b99c.slice/crio-6e7f126945ae9eb6cdce75006b3cb8504d0949bbd20a5b6d974a4aff4bb44a7e WatchSource:0}: Error finding container 6e7f126945ae9eb6cdce75006b3cb8504d0949bbd20a5b6d974a4aff4bb44a7e: Status 404 returned error can't find the container with id 6e7f126945ae9eb6cdce75006b3cb8504d0949bbd20a5b6d974a4aff4bb44a7e
Sep 05 20:01:09.635680 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:09.635591    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt"]
Sep 05 20:01:10.012101 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:10.011987    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9" event={"ID":"9a94483b-1654-4f71-b48d-869d3a0f36e4","Type":"ContainerStarted","Data":"3deeece6cd34c0164474863d3857c16e74e581f2ef7aa372b2f1ae7abcfedd59"}
Sep 05 20:01:10.012101 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:10.012063    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9" event={"ID":"9a94483b-1654-4f71-b48d-869d3a0f36e4","Type":"ContainerStarted","Data":"98e007bb5b377ec3c0a290baa2b5e335b2d1eab73a8aae62df6350083532af95"}
Sep 05 20:01:10.012648 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:10.012208    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9"
Sep 05 20:01:10.014157 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:10.014051    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt" event={"ID":"61ced69f-fa9e-4a79-9a66-dfc7d276b99c","Type":"ContainerStarted","Data":"3f88650f332fe7c8936c153a7068e4668cc314e2f9820d9adf64fe1b7623dd42"}
Sep 05 20:01:10.014157 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:10.014100    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt" event={"ID":"61ced69f-fa9e-4a79-9a66-dfc7d276b99c","Type":"ContainerStarted","Data":"6e7f126945ae9eb6cdce75006b3cb8504d0949bbd20a5b6d974a4aff4bb44a7e"}
Sep 05 20:01:10.014361 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:10.014172    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt"
Sep 05 20:01:10.031796 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:10.031735    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9" podStartSLOduration=2.031721782 podStartE2EDuration="2.031721782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:01:10.030478836 +0000 UTC m=+1069.060110742" watchObservedRunningTime="2026-09-05 20:01:10.031721782 +0000 UTC m=+1069.061353687"
Sep 05 20:01:10.046973 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:10.046919    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt" podStartSLOduration=2.046903869 podStartE2EDuration="2.046903869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:01:10.045902564 +0000 UTC m=+1069.075534471" watchObservedRunningTime="2026-09-05 20:01:10.046903869 +0000 UTC m=+1069.076535816"
Sep 05 20:01:12.275387 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:12.275349    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1fd18b-29810640-zv4zk"]
Sep 05 20:01:12.280276 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:12.280244    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-1fd18b-29810640-zv4zk"]
Sep 05 20:01:12.301993 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:12.301959    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb"]
Sep 05 20:01:12.302240 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:12.302213    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb" podUID="7c2bab11-ce91-4eea-9a5d-47f8aff048f3" containerName="payload-processing" containerID="cri-o://acb1ca7e382c6d46092935bbc3247da49ee9c5ca3a2228d06ec1cbed2a01b669" gracePeriod=30
Sep 05 20:01:12.462216 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:12.462182    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1fd18b-wbsz5"]
Sep 05 20:01:12.472794 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:12.470182    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-1fd18b-wbsz5"]
Sep 05 20:01:12.583959 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:12.583934    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb"
Sep 05 20:01:12.726083 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:12.726047    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c2bab11-ce91-4eea-9a5d-47f8aff048f3-plugins-config-volume\") pod \"7c2bab11-ce91-4eea-9a5d-47f8aff048f3\" (UID: \"7c2bab11-ce91-4eea-9a5d-47f8aff048f3\") "
Sep 05 20:01:12.726295 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:12.726158    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-296f5\" (UniqueName: \"kubernetes.io/projected/7c2bab11-ce91-4eea-9a5d-47f8aff048f3-kube-api-access-296f5\") pod \"7c2bab11-ce91-4eea-9a5d-47f8aff048f3\" (UID: \"7c2bab11-ce91-4eea-9a5d-47f8aff048f3\") "
Sep 05 20:01:12.726488 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:12.726458    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c2bab11-ce91-4eea-9a5d-47f8aff048f3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7c2bab11-ce91-4eea-9a5d-47f8aff048f3" (UID: "7c2bab11-ce91-4eea-9a5d-47f8aff048f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:01:12.728240 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:12.728208    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2bab11-ce91-4eea-9a5d-47f8aff048f3-kube-api-access-296f5" (OuterVolumeSpecName: "kube-api-access-296f5") pod "7c2bab11-ce91-4eea-9a5d-47f8aff048f3" (UID: "7c2bab11-ce91-4eea-9a5d-47f8aff048f3"). InnerVolumeSpecName "kube-api-access-296f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:01:12.827653 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:12.827623    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c2bab11-ce91-4eea-9a5d-47f8aff048f3-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:01:12.827653 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:12.827652    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-296f5\" (UniqueName: \"kubernetes.io/projected/7c2bab11-ce91-4eea-9a5d-47f8aff048f3-kube-api-access-296f5\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:01:13.036279 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:13.036245    2583 generic.go:356] "Generic (PLEG): container finished" podID="7c2bab11-ce91-4eea-9a5d-47f8aff048f3" containerID="acb1ca7e382c6d46092935bbc3247da49ee9c5ca3a2228d06ec1cbed2a01b669" exitCode=0
Sep 05 20:01:13.036451 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:13.036288    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb"
Sep 05 20:01:13.036451 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:13.036332    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb" event={"ID":"7c2bab11-ce91-4eea-9a5d-47f8aff048f3","Type":"ContainerDied","Data":"acb1ca7e382c6d46092935bbc3247da49ee9c5ca3a2228d06ec1cbed2a01b669"}
Sep 05 20:01:13.036451 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:13.036395    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb" event={"ID":"7c2bab11-ce91-4eea-9a5d-47f8aff048f3","Type":"ContainerDied","Data":"ed4d9319deba39dd3f1868034060d8b2cea98ac500881262c8be5a0e2225ed13"}
Sep 05 20:01:13.036451 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:13.036424    2583 scope.go:117] "RemoveContainer" containerID="acb1ca7e382c6d46092935bbc3247da49ee9c5ca3a2228d06ec1cbed2a01b669"
Sep 05 20:01:13.045984 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:13.045963    2583 scope.go:117] "RemoveContainer" containerID="acb1ca7e382c6d46092935bbc3247da49ee9c5ca3a2228d06ec1cbed2a01b669"
Sep 05 20:01:13.046257 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:01:13.046240    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acb1ca7e382c6d46092935bbc3247da49ee9c5ca3a2228d06ec1cbed2a01b669\": container with ID starting with acb1ca7e382c6d46092935bbc3247da49ee9c5ca3a2228d06ec1cbed2a01b669 not found: ID does not exist" containerID="acb1ca7e382c6d46092935bbc3247da49ee9c5ca3a2228d06ec1cbed2a01b669"
Sep 05 20:01:13.046306 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:13.046266    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acb1ca7e382c6d46092935bbc3247da49ee9c5ca3a2228d06ec1cbed2a01b669"} err="failed to get container status \"acb1ca7e382c6d46092935bbc3247da49ee9c5ca3a2228d06ec1cbed2a01b669\": rpc error: code = NotFound desc = could not find container \"acb1ca7e382c6d46092935bbc3247da49ee9c5ca3a2228d06ec1cbed2a01b669\": container with ID starting with acb1ca7e382c6d46092935bbc3247da49ee9c5ca3a2228d06ec1cbed2a01b669 not found: ID does not exist"
Sep 05 20:01:13.061633 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:13.061602    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb"]
Sep 05 20:01:13.070635 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:13.070605    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-1fd18b-7fc6f8c64f-rhvtb"]
Sep 05 20:01:13.436566 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:13.436531    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c2bab11-ce91-4eea-9a5d-47f8aff048f3" path="/var/lib/kubelet/pods/7c2bab11-ce91-4eea-9a5d-47f8aff048f3/volumes"
Sep 05 20:01:13.436957 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:13.436936    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0167081-17d9-4e12-aa99-16e2b0aeb582" path="/var/lib/kubelet/pods/b0167081-17d9-4e12-aa99-16e2b0aeb582/volumes"
Sep 05 20:01:13.437243 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:13.437228    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8270f44-b821-4b4a-afab-0638b8b5c210" path="/var/lib/kubelet/pods/c8270f44-b821-4b4a-afab-0638b8b5c210/volumes"
Sep 05 20:01:20.729506 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:20.729429    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq" podUID="b57496d8-319a-4506-9a23-554147cb8329" containerName="main" containerID="cri-o://df4e5551c2d4528ec0f96b4223aa23c113a1b489946328cfdebde89e29d64b8e" gracePeriod=45
Sep 05 20:01:20.988231 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:20.988174    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:01:21.025000 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.024968    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9"
Sep 05 20:01:21.025204 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.025044    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt"
Sep 05 20:01:21.091206 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.091171    2583 generic.go:356] "Generic (PLEG): container finished" podID="b57496d8-319a-4506-9a23-554147cb8329" containerID="df4e5551c2d4528ec0f96b4223aa23c113a1b489946328cfdebde89e29d64b8e" exitCode=0
Sep 05 20:01:21.091360 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.091216    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"
Sep 05 20:01:21.091360 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.091253    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq" event={"ID":"b57496d8-319a-4506-9a23-554147cb8329","Type":"ContainerDied","Data":"df4e5551c2d4528ec0f96b4223aa23c113a1b489946328cfdebde89e29d64b8e"}
Sep 05 20:01:21.091360 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.091297    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq" event={"ID":"b57496d8-319a-4506-9a23-554147cb8329","Type":"ContainerDied","Data":"5339d18bcd786a2ec505763af35008634a1ed28b9037d7c6cbdebaccef25efd4"}
Sep 05 20:01:21.091360 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.091315    2583 scope.go:117] "RemoveContainer" containerID="df4e5551c2d4528ec0f96b4223aa23c113a1b489946328cfdebde89e29d64b8e"
Sep 05 20:01:21.101040 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.101021    2583 scope.go:117] "RemoveContainer" containerID="df4e5551c2d4528ec0f96b4223aa23c113a1b489946328cfdebde89e29d64b8e"
Sep 05 20:01:21.101419 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:01:21.101390    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df4e5551c2d4528ec0f96b4223aa23c113a1b489946328cfdebde89e29d64b8e\": container with ID starting with df4e5551c2d4528ec0f96b4223aa23c113a1b489946328cfdebde89e29d64b8e not found: ID does not exist" containerID="df4e5551c2d4528ec0f96b4223aa23c113a1b489946328cfdebde89e29d64b8e"
Sep 05 20:01:21.101510 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.101432    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df4e5551c2d4528ec0f96b4223aa23c113a1b489946328cfdebde89e29d64b8e"} err="failed to get container status \"df4e5551c2d4528ec0f96b4223aa23c113a1b489946328cfdebde89e29d64b8e\": rpc error: code = NotFound desc = could not find container \"df4e5551c2d4528ec0f96b4223aa23c113a1b489946328cfdebde89e29d64b8e\": container with ID starting with df4e5551c2d4528ec0f96b4223aa23c113a1b489946328cfdebde89e29d64b8e not found: ID does not exist"
Sep 05 20:01:21.106795 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.106773    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b57496d8-319a-4506-9a23-554147cb8329-tmp-dir\") pod \"b57496d8-319a-4506-9a23-554147cb8329\" (UID: \"b57496d8-319a-4506-9a23-554147cb8329\") "
Sep 05 20:01:21.106888 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.106815    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b57496d8-319a-4506-9a23-554147cb8329-home\") pod \"b57496d8-319a-4506-9a23-554147cb8329\" (UID: \"b57496d8-319a-4506-9a23-554147cb8329\") "
Sep 05 20:01:21.106888 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.106843    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b57496d8-319a-4506-9a23-554147cb8329-dshm\") pod \"b57496d8-319a-4506-9a23-554147cb8329\" (UID: \"b57496d8-319a-4506-9a23-554147cb8329\") "
Sep 05 20:01:21.106888 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.106876    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b57496d8-319a-4506-9a23-554147cb8329-tls-certs\") pod \"b57496d8-319a-4506-9a23-554147cb8329\" (UID: \"b57496d8-319a-4506-9a23-554147cb8329\") "
Sep 05 20:01:21.107058 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.106919    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rkjt\" (UniqueName: \"kubernetes.io/projected/b57496d8-319a-4506-9a23-554147cb8329-kube-api-access-2rkjt\") pod \"b57496d8-319a-4506-9a23-554147cb8329\" (UID: \"b57496d8-319a-4506-9a23-554147cb8329\") "
Sep 05 20:01:21.107058 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.106988    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b57496d8-319a-4506-9a23-554147cb8329-model-cache\") pod \"b57496d8-319a-4506-9a23-554147cb8329\" (UID: \"b57496d8-319a-4506-9a23-554147cb8329\") "
Sep 05 20:01:21.107058 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.106988    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57496d8-319a-4506-9a23-554147cb8329-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b57496d8-319a-4506-9a23-554147cb8329" (UID: "b57496d8-319a-4506-9a23-554147cb8329"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:01:21.107366 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.107337    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57496d8-319a-4506-9a23-554147cb8329-home" (OuterVolumeSpecName: "home") pod "b57496d8-319a-4506-9a23-554147cb8329" (UID: "b57496d8-319a-4506-9a23-554147cb8329"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:01:21.107464 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.107397    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b57496d8-319a-4506-9a23-554147cb8329-tmp-dir\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:01:21.107561 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.107525    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57496d8-319a-4506-9a23-554147cb8329-model-cache" (OuterVolumeSpecName: "model-cache") pod "b57496d8-319a-4506-9a23-554147cb8329" (UID: "b57496d8-319a-4506-9a23-554147cb8329"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:01:21.109208 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.109178    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b57496d8-319a-4506-9a23-554147cb8329-dshm" (OuterVolumeSpecName: "dshm") pod "b57496d8-319a-4506-9a23-554147cb8329" (UID: "b57496d8-319a-4506-9a23-554147cb8329"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:01:21.109313 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.109255    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b57496d8-319a-4506-9a23-554147cb8329-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b57496d8-319a-4506-9a23-554147cb8329" (UID: "b57496d8-319a-4506-9a23-554147cb8329"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:01:21.109313 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.109277    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b57496d8-319a-4506-9a23-554147cb8329-kube-api-access-2rkjt" (OuterVolumeSpecName: "kube-api-access-2rkjt") pod "b57496d8-319a-4506-9a23-554147cb8329" (UID: "b57496d8-319a-4506-9a23-554147cb8329"). InnerVolumeSpecName "kube-api-access-2rkjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:01:21.207972 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.207942    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b57496d8-319a-4506-9a23-554147cb8329-home\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:01:21.207972 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.207967    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b57496d8-319a-4506-9a23-554147cb8329-dshm\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:01:21.207972 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.207975    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b57496d8-319a-4506-9a23-554147cb8329-tls-certs\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:01:21.208322 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.207985    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rkjt\" (UniqueName: \"kubernetes.io/projected/b57496d8-319a-4506-9a23-554147cb8329-kube-api-access-2rkjt\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:01:21.208322 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.207994    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b57496d8-319a-4506-9a23-554147cb8329-model-cache\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:01:21.413993 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.413962    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"]
Sep 05 20:01:21.419897 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.419873    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-c038f394-kserve-6d7fd6f449-874dq"]
Sep 05 20:01:21.435407 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:21.435380    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b57496d8-319a-4506-9a23-554147cb8329" path="/var/lib/kubelet/pods/b57496d8-319a-4506-9a23-554147cb8329/volumes"
Sep 05 20:01:38.869095 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:38.869058    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt"]
Sep 05 20:01:38.870039 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:38.869706    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt" podUID="61ced69f-fa9e-4a79-9a66-dfc7d276b99c" containerName="payload-processing" containerID="cri-o://3f88650f332fe7c8936c153a7068e4668cc314e2f9820d9adf64fe1b7623dd42" gracePeriod=30
Sep 05 20:01:38.874340 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:38.874292    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9"]
Sep 05 20:01:38.875014 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:38.874525    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9" podUID="9a94483b-1654-4f71-b48d-869d3a0f36e4" containerName="payload-pre-processing" containerID="cri-o://3deeece6cd34c0164474863d3857c16e74e581f2ef7aa372b2f1ae7abcfedd59" gracePeriod=30
Sep 05 20:01:39.177059 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.177031    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt"
Sep 05 20:01:39.180358 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.180334    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9"
Sep 05 20:01:39.220014 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.219980    2583 generic.go:356] "Generic (PLEG): container finished" podID="9a94483b-1654-4f71-b48d-869d3a0f36e4" containerID="3deeece6cd34c0164474863d3857c16e74e581f2ef7aa372b2f1ae7abcfedd59" exitCode=0
Sep 05 20:01:39.220214 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.220033    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9"
Sep 05 20:01:39.220214 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.220068    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9" event={"ID":"9a94483b-1654-4f71-b48d-869d3a0f36e4","Type":"ContainerDied","Data":"3deeece6cd34c0164474863d3857c16e74e581f2ef7aa372b2f1ae7abcfedd59"}
Sep 05 20:01:39.220214 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.220139    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9" event={"ID":"9a94483b-1654-4f71-b48d-869d3a0f36e4","Type":"ContainerDied","Data":"98e007bb5b377ec3c0a290baa2b5e335b2d1eab73a8aae62df6350083532af95"}
Sep 05 20:01:39.220214 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.220171    2583 scope.go:117] "RemoveContainer" containerID="3deeece6cd34c0164474863d3857c16e74e581f2ef7aa372b2f1ae7abcfedd59"
Sep 05 20:01:39.221796 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.221765    2583 generic.go:356] "Generic (PLEG): container finished" podID="61ced69f-fa9e-4a79-9a66-dfc7d276b99c" containerID="3f88650f332fe7c8936c153a7068e4668cc314e2f9820d9adf64fe1b7623dd42" exitCode=0
Sep 05 20:01:39.221951 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.221813    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt"
Sep 05 20:01:39.221951 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.221821    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt" event={"ID":"61ced69f-fa9e-4a79-9a66-dfc7d276b99c","Type":"ContainerDied","Data":"3f88650f332fe7c8936c153a7068e4668cc314e2f9820d9adf64fe1b7623dd42"}
Sep 05 20:01:39.221951 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.221862    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt" event={"ID":"61ced69f-fa9e-4a79-9a66-dfc7d276b99c","Type":"ContainerDied","Data":"6e7f126945ae9eb6cdce75006b3cb8504d0949bbd20a5b6d974a4aff4bb44a7e"}
Sep 05 20:01:39.231170 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.231154    2583 scope.go:117] "RemoveContainer" containerID="3deeece6cd34c0164474863d3857c16e74e581f2ef7aa372b2f1ae7abcfedd59"
Sep 05 20:01:39.231488 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:01:39.231467    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3deeece6cd34c0164474863d3857c16e74e581f2ef7aa372b2f1ae7abcfedd59\": container with ID starting with 3deeece6cd34c0164474863d3857c16e74e581f2ef7aa372b2f1ae7abcfedd59 not found: ID does not exist" containerID="3deeece6cd34c0164474863d3857c16e74e581f2ef7aa372b2f1ae7abcfedd59"
Sep 05 20:01:39.231559 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.231492    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3deeece6cd34c0164474863d3857c16e74e581f2ef7aa372b2f1ae7abcfedd59"} err="failed to get container status \"3deeece6cd34c0164474863d3857c16e74e581f2ef7aa372b2f1ae7abcfedd59\": rpc error: code = NotFound desc = could not find container \"3deeece6cd34c0164474863d3857c16e74e581f2ef7aa372b2f1ae7abcfedd59\": container with ID starting with 3deeece6cd34c0164474863d3857c16e74e581f2ef7aa372b2f1ae7abcfedd59 not found: ID does not exist"
Sep 05 20:01:39.231559 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.231513    2583 scope.go:117] "RemoveContainer" containerID="3f88650f332fe7c8936c153a7068e4668cc314e2f9820d9adf64fe1b7623dd42"
Sep 05 20:01:39.240482 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.240461    2583 scope.go:117] "RemoveContainer" containerID="3f88650f332fe7c8936c153a7068e4668cc314e2f9820d9adf64fe1b7623dd42"
Sep 05 20:01:39.240811 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:01:39.240788    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f88650f332fe7c8936c153a7068e4668cc314e2f9820d9adf64fe1b7623dd42\": container with ID starting with 3f88650f332fe7c8936c153a7068e4668cc314e2f9820d9adf64fe1b7623dd42 not found: ID does not exist" containerID="3f88650f332fe7c8936c153a7068e4668cc314e2f9820d9adf64fe1b7623dd42"
Sep 05 20:01:39.240926 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.240813    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f88650f332fe7c8936c153a7068e4668cc314e2f9820d9adf64fe1b7623dd42"} err="failed to get container status \"3f88650f332fe7c8936c153a7068e4668cc314e2f9820d9adf64fe1b7623dd42\": rpc error: code = NotFound desc = could not find container \"3f88650f332fe7c8936c153a7068e4668cc314e2f9820d9adf64fe1b7623dd42\": container with ID starting with 3f88650f332fe7c8936c153a7068e4668cc314e2f9820d9adf64fe1b7623dd42 not found: ID does not exist"
Sep 05 20:01:39.282580 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.282543    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgdxw\" (UniqueName: \"kubernetes.io/projected/61ced69f-fa9e-4a79-9a66-dfc7d276b99c-kube-api-access-vgdxw\") pod \"61ced69f-fa9e-4a79-9a66-dfc7d276b99c\" (UID: \"61ced69f-fa9e-4a79-9a66-dfc7d276b99c\") "
Sep 05 20:01:39.282792 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.282619    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61ced69f-fa9e-4a79-9a66-dfc7d276b99c-plugins-config-volume\") pod \"61ced69f-fa9e-4a79-9a66-dfc7d276b99c\" (UID: \"61ced69f-fa9e-4a79-9a66-dfc7d276b99c\") "
Sep 05 20:01:39.282792 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.282651    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4vkk\" (UniqueName: \"kubernetes.io/projected/9a94483b-1654-4f71-b48d-869d3a0f36e4-kube-api-access-g4vkk\") pod \"9a94483b-1654-4f71-b48d-869d3a0f36e4\" (UID: \"9a94483b-1654-4f71-b48d-869d3a0f36e4\") "
Sep 05 20:01:39.282792 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.282747    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a94483b-1654-4f71-b48d-869d3a0f36e4-plugins-config-volume\") pod \"9a94483b-1654-4f71-b48d-869d3a0f36e4\" (UID: \"9a94483b-1654-4f71-b48d-869d3a0f36e4\") "
Sep 05 20:01:39.283486 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.283453    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a94483b-1654-4f71-b48d-869d3a0f36e4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9a94483b-1654-4f71-b48d-869d3a0f36e4" (UID: "9a94483b-1654-4f71-b48d-869d3a0f36e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:01:39.283646 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.283605    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61ced69f-fa9e-4a79-9a66-dfc7d276b99c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "61ced69f-fa9e-4a79-9a66-dfc7d276b99c" (UID: "61ced69f-fa9e-4a79-9a66-dfc7d276b99c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:01:39.285818 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.285794    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ced69f-fa9e-4a79-9a66-dfc7d276b99c-kube-api-access-vgdxw" (OuterVolumeSpecName: "kube-api-access-vgdxw") pod "61ced69f-fa9e-4a79-9a66-dfc7d276b99c" (UID: "61ced69f-fa9e-4a79-9a66-dfc7d276b99c"). InnerVolumeSpecName "kube-api-access-vgdxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:01:39.285908 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.285840    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a94483b-1654-4f71-b48d-869d3a0f36e4-kube-api-access-g4vkk" (OuterVolumeSpecName: "kube-api-access-g4vkk") pod "9a94483b-1654-4f71-b48d-869d3a0f36e4" (UID: "9a94483b-1654-4f71-b48d-869d3a0f36e4"). InnerVolumeSpecName "kube-api-access-g4vkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:01:39.384533 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.384501    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a94483b-1654-4f71-b48d-869d3a0f36e4-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:01:39.384533 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.384528    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgdxw\" (UniqueName: \"kubernetes.io/projected/61ced69f-fa9e-4a79-9a66-dfc7d276b99c-kube-api-access-vgdxw\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:01:39.384533 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.384538    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61ced69f-fa9e-4a79-9a66-dfc7d276b99c-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:01:39.384761 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.384546    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4vkk\" (UniqueName: \"kubernetes.io/projected/9a94483b-1654-4f71-b48d-869d3a0f36e4-kube-api-access-g4vkk\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:01:39.540603 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.540568    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9"]
Sep 05 20:01:39.544404 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.544372    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c06406ce-5dcc695c46-wjhv9"]
Sep 05 20:01:39.554065 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.554035    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt"]
Sep 05 20:01:39.558237 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:39.558215    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c06406ce-79dd8b9db5-7knrt"]
Sep 05 20:01:41.435277 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:41.435243    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61ced69f-fa9e-4a79-9a66-dfc7d276b99c" path="/var/lib/kubelet/pods/61ced69f-fa9e-4a79-9a66-dfc7d276b99c/volumes"
Sep 05 20:01:41.435636 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:41.435578    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a94483b-1654-4f71-b48d-869d3a0f36e4" path="/var/lib/kubelet/pods/9a94483b-1654-4f71-b48d-869d3a0f36e4/volumes"
Sep 05 20:01:47.250540 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.250505    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"]
Sep 05 20:01:47.250930 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.250910    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61ced69f-fa9e-4a79-9a66-dfc7d276b99c" containerName="payload-processing"
Sep 05 20:01:47.250930 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.250921    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="61ced69f-fa9e-4a79-9a66-dfc7d276b99c" containerName="payload-processing"
Sep 05 20:01:47.251027 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.250934    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b57496d8-319a-4506-9a23-554147cb8329" containerName="main"
Sep 05 20:01:47.251027 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.250941    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b57496d8-319a-4506-9a23-554147cb8329" containerName="main"
Sep 05 20:01:47.251027 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.250959    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a94483b-1654-4f71-b48d-869d3a0f36e4" containerName="payload-pre-processing"
Sep 05 20:01:47.251027 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.250968    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a94483b-1654-4f71-b48d-869d3a0f36e4" containerName="payload-pre-processing"
Sep 05 20:01:47.251027 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.250983    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c2bab11-ce91-4eea-9a5d-47f8aff048f3" containerName="payload-processing"
Sep 05 20:01:47.251027 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.250988    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c2bab11-ce91-4eea-9a5d-47f8aff048f3" containerName="payload-processing"
Sep 05 20:01:47.251278 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.251060    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="61ced69f-fa9e-4a79-9a66-dfc7d276b99c" containerName="payload-processing"
Sep 05 20:01:47.251278 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.251073    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c2bab11-ce91-4eea-9a5d-47f8aff048f3" containerName="payload-processing"
Sep 05 20:01:47.251278 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.251086    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b57496d8-319a-4506-9a23-554147cb8329" containerName="main"
Sep 05 20:01:47.251278 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.251091    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a94483b-1654-4f71-b48d-869d3a0f36e4" containerName="payload-pre-processing"
Sep 05 20:01:47.255712 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.255692    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:01:47.257859 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.257834    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1dc7b1\"/\"openshift-service-ca.crt\""
Sep 05 20:01:47.258411 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.258393    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1dc7b1\"/\"kube-root-ca.crt\""
Sep 05 20:01:47.258517 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.258465    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1dc7b1\"/\"test-model-1dc7b1-kserve-self-signed-certs\""
Sep 05 20:01:47.258581 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.258525    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1dc7b1\"/\"default-dockercfg-gfvnf\""
Sep 05 20:01:47.265389 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.265361    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"]
Sep 05 20:01:47.355365 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.355331    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8687da3a-7a11-4a9f-9f72-95dc214f413f-tls-certs\") pod \"test-model-1dc7b1-kserve-759fbc96b9-6cnwq\" (UID: \"8687da3a-7a11-4a9f-9f72-95dc214f413f\") " pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:01:47.355518 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.355378    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8687da3a-7a11-4a9f-9f72-95dc214f413f-dshm\") pod \"test-model-1dc7b1-kserve-759fbc96b9-6cnwq\" (UID: \"8687da3a-7a11-4a9f-9f72-95dc214f413f\") " pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:01:47.355518 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.355404    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8687da3a-7a11-4a9f-9f72-95dc214f413f-home\") pod \"test-model-1dc7b1-kserve-759fbc96b9-6cnwq\" (UID: \"8687da3a-7a11-4a9f-9f72-95dc214f413f\") " pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:01:47.355518 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.355431    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pw7q\" (UniqueName: \"kubernetes.io/projected/8687da3a-7a11-4a9f-9f72-95dc214f413f-kube-api-access-6pw7q\") pod \"test-model-1dc7b1-kserve-759fbc96b9-6cnwq\" (UID: \"8687da3a-7a11-4a9f-9f72-95dc214f413f\") " pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:01:47.355518 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.355470    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8687da3a-7a11-4a9f-9f72-95dc214f413f-tmp-dir\") pod \"test-model-1dc7b1-kserve-759fbc96b9-6cnwq\" (UID: \"8687da3a-7a11-4a9f-9f72-95dc214f413f\") " pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:01:47.355656 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.355601    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8687da3a-7a11-4a9f-9f72-95dc214f413f-model-cache\") pod \"test-model-1dc7b1-kserve-759fbc96b9-6cnwq\" (UID: \"8687da3a-7a11-4a9f-9f72-95dc214f413f\") " pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:01:47.456700 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.456670    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8687da3a-7a11-4a9f-9f72-95dc214f413f-tmp-dir\") pod \"test-model-1dc7b1-kserve-759fbc96b9-6cnwq\" (UID: \"8687da3a-7a11-4a9f-9f72-95dc214f413f\") " pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:01:47.456892 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.456775    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8687da3a-7a11-4a9f-9f72-95dc214f413f-model-cache\") pod \"test-model-1dc7b1-kserve-759fbc96b9-6cnwq\" (UID: \"8687da3a-7a11-4a9f-9f72-95dc214f413f\") " pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:01:47.456892 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.456802    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8687da3a-7a11-4a9f-9f72-95dc214f413f-tls-certs\") pod \"test-model-1dc7b1-kserve-759fbc96b9-6cnwq\" (UID: \"8687da3a-7a11-4a9f-9f72-95dc214f413f\") " pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:01:47.456892 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.456842    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8687da3a-7a11-4a9f-9f72-95dc214f413f-dshm\") pod \"test-model-1dc7b1-kserve-759fbc96b9-6cnwq\" (UID: \"8687da3a-7a11-4a9f-9f72-95dc214f413f\") " pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:01:47.456892 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.456874    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8687da3a-7a11-4a9f-9f72-95dc214f413f-home\") pod \"test-model-1dc7b1-kserve-759fbc96b9-6cnwq\" (UID: \"8687da3a-7a11-4a9f-9f72-95dc214f413f\") " pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:01:47.457142 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.456902    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pw7q\" (UniqueName: \"kubernetes.io/projected/8687da3a-7a11-4a9f-9f72-95dc214f413f-kube-api-access-6pw7q\") pod \"test-model-1dc7b1-kserve-759fbc96b9-6cnwq\" (UID: \"8687da3a-7a11-4a9f-9f72-95dc214f413f\") " pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:01:47.457142 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.457080    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8687da3a-7a11-4a9f-9f72-95dc214f413f-tmp-dir\") pod \"test-model-1dc7b1-kserve-759fbc96b9-6cnwq\" (UID: \"8687da3a-7a11-4a9f-9f72-95dc214f413f\") " pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:01:47.457267 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.457155    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8687da3a-7a11-4a9f-9f72-95dc214f413f-model-cache\") pod \"test-model-1dc7b1-kserve-759fbc96b9-6cnwq\" (UID: \"8687da3a-7a11-4a9f-9f72-95dc214f413f\") " pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:01:47.457267 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.457239    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8687da3a-7a11-4a9f-9f72-95dc214f413f-home\") pod \"test-model-1dc7b1-kserve-759fbc96b9-6cnwq\" (UID: \"8687da3a-7a11-4a9f-9f72-95dc214f413f\") " pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:01:47.459139 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.459091    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8687da3a-7a11-4a9f-9f72-95dc214f413f-dshm\") pod \"test-model-1dc7b1-kserve-759fbc96b9-6cnwq\" (UID: \"8687da3a-7a11-4a9f-9f72-95dc214f413f\") " pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:01:47.459361 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.459341    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8687da3a-7a11-4a9f-9f72-95dc214f413f-tls-certs\") pod \"test-model-1dc7b1-kserve-759fbc96b9-6cnwq\" (UID: \"8687da3a-7a11-4a9f-9f72-95dc214f413f\") " pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:01:47.466329 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.466293    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pw7q\" (UniqueName: \"kubernetes.io/projected/8687da3a-7a11-4a9f-9f72-95dc214f413f-kube-api-access-6pw7q\") pod \"test-model-1dc7b1-kserve-759fbc96b9-6cnwq\" (UID: \"8687da3a-7a11-4a9f-9f72-95dc214f413f\") " pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:01:47.567254 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.567174    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:01:47.843360 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:47.843332    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"]
Sep 05 20:01:47.844788 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:01:47.844752    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8687da3a_7a11_4a9f_9f72_95dc214f413f.slice/crio-83331a919e2bbc0327004c9d33841566816c94c539d54e9f57ddd4541440750c WatchSource:0}: Error finding container 83331a919e2bbc0327004c9d33841566816c94c539d54e9f57ddd4541440750c: Status 404 returned error can't find the container with id 83331a919e2bbc0327004c9d33841566816c94c539d54e9f57ddd4541440750c
Sep 05 20:01:48.283136 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:48.283025    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq" event={"ID":"8687da3a-7a11-4a9f-9f72-95dc214f413f","Type":"ContainerStarted","Data":"2989a97ab2f25fbe86a91b87e6b80c173fd615619993b2d62456c460544b6e8f"}
Sep 05 20:01:48.283136 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:48.283070    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq" event={"ID":"8687da3a-7a11-4a9f-9f72-95dc214f413f","Type":"ContainerStarted","Data":"83331a919e2bbc0327004c9d33841566816c94c539d54e9f57ddd4541440750c"}
Sep 05 20:01:48.302419 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:48.302369    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq" podStartSLOduration=1.302352814 podStartE2EDuration="1.302352814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:01:48.300570521 +0000 UTC m=+1107.330202428" watchObservedRunningTime="2026-09-05 20:01:48.302352814 +0000 UTC m=+1107.331984719"
Sep 05 20:01:55.079894 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:55.079856    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"]
Sep 05 20:01:55.080317 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:55.080226    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs" podUID="8b3cd79b-acbb-415f-bda7-43d0d4419620" containerName="istio-proxy" containerID="cri-o://1d71e43a9a81717091e294cd35c779cb2fe804761d212a8666764da132649e66" gracePeriod=30
Sep 05 20:01:57.567477 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:57.567439    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:01:57.567942 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:57.567490    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:01:57.581270 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:57.581244    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:01:58.360051 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:01:58.360023    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:02:00.355511 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.355448    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 20:02:00.362431 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.362403    2583 generic.go:356] "Generic (PLEG): container finished" podID="8b3cd79b-acbb-415f-bda7-43d0d4419620" containerID="1d71e43a9a81717091e294cd35c779cb2fe804761d212a8666764da132649e66" exitCode=0
Sep 05 20:02:00.362565 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.362449    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"
Sep 05 20:02:00.362565 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.362447    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs" event={"ID":"8b3cd79b-acbb-415f-bda7-43d0d4419620","Type":"ContainerDied","Data":"1d71e43a9a81717091e294cd35c779cb2fe804761d212a8666764da132649e66"}
Sep 05 20:02:00.362565 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.362500    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs" event={"ID":"8b3cd79b-acbb-415f-bda7-43d0d4419620","Type":"ContainerDied","Data":"f7d6a4bc66bf97f38fc9fa3a12e555c6b2493ddce9f953e2416e2792fc022c83"}
Sep 05 20:02:00.362565 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.362523    2583 scope.go:117] "RemoveContainer" containerID="1d71e43a9a81717091e294cd35c779cb2fe804761d212a8666764da132649e66"
Sep 05 20:02:00.374734 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.374713    2583 scope.go:117] "RemoveContainer" containerID="0781b43fded62be76f1965d2bd275a48551dd64b84d96c37e1b0378b448ad02a"
Sep 05 20:02:00.384712 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.384694    2583 scope.go:117] "RemoveContainer" containerID="1d71e43a9a81717091e294cd35c779cb2fe804761d212a8666764da132649e66"
Sep 05 20:02:00.385034 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:02:00.385012    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d71e43a9a81717091e294cd35c779cb2fe804761d212a8666764da132649e66\": container with ID starting with 1d71e43a9a81717091e294cd35c779cb2fe804761d212a8666764da132649e66 not found: ID does not exist" containerID="1d71e43a9a81717091e294cd35c779cb2fe804761d212a8666764da132649e66"
Sep 05 20:02:00.385145 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.385044    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d71e43a9a81717091e294cd35c779cb2fe804761d212a8666764da132649e66"} err="failed to get container status \"1d71e43a9a81717091e294cd35c779cb2fe804761d212a8666764da132649e66\": rpc error: code = NotFound desc = could not find container \"1d71e43a9a81717091e294cd35c779cb2fe804761d212a8666764da132649e66\": container with ID starting with 1d71e43a9a81717091e294cd35c779cb2fe804761d212a8666764da132649e66 not found: ID does not exist"
Sep 05 20:02:00.385145 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.385071    2583 scope.go:117] "RemoveContainer" containerID="0781b43fded62be76f1965d2bd275a48551dd64b84d96c37e1b0378b448ad02a"
Sep 05 20:02:00.385468 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:02:00.385443    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0781b43fded62be76f1965d2bd275a48551dd64b84d96c37e1b0378b448ad02a\": container with ID starting with 0781b43fded62be76f1965d2bd275a48551dd64b84d96c37e1b0378b448ad02a not found: ID does not exist" containerID="0781b43fded62be76f1965d2bd275a48551dd64b84d96c37e1b0378b448ad02a"
Sep 05 20:02:00.385511 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.385478    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0781b43fded62be76f1965d2bd275a48551dd64b84d96c37e1b0378b448ad02a"} err="failed to get container status \"0781b43fded62be76f1965d2bd275a48551dd64b84d96c37e1b0378b448ad02a\": rpc error: code = NotFound desc = could not find container \"0781b43fded62be76f1965d2bd275a48551dd64b84d96c37e1b0378b448ad02a\": container with ID starting with 0781b43fded62be76f1965d2bd275a48551dd64b84d96c37e1b0378b448ad02a not found: ID does not exist"
Sep 05 20:02:00.490766 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.490736    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jtvg\" (UniqueName: \"kubernetes.io/projected/8b3cd79b-acbb-415f-bda7-43d0d4419620-kube-api-access-7jtvg\") pod \"8b3cd79b-acbb-415f-bda7-43d0d4419620\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") "
Sep 05 20:02:00.490959 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.490788    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b3cd79b-acbb-415f-bda7-43d0d4419620-istiod-ca-cert\") pod \"8b3cd79b-acbb-415f-bda7-43d0d4419620\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") "
Sep 05 20:02:00.490959 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.490812    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b3cd79b-acbb-415f-bda7-43d0d4419620-istio-token\") pod \"8b3cd79b-acbb-415f-bda7-43d0d4419620\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") "
Sep 05 20:02:00.490959 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.490872    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b3cd79b-acbb-415f-bda7-43d0d4419620-istio-podinfo\") pod \"8b3cd79b-acbb-415f-bda7-43d0d4419620\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") "
Sep 05 20:02:00.490959 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.490916    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-istio-data\") pod \"8b3cd79b-acbb-415f-bda7-43d0d4419620\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") "
Sep 05 20:02:00.491266 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.491035    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-istio-envoy\") pod \"8b3cd79b-acbb-415f-bda7-43d0d4419620\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") "
Sep 05 20:02:00.491266 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.491167    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-credential-socket\") pod \"8b3cd79b-acbb-415f-bda7-43d0d4419620\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") "
Sep 05 20:02:00.491266 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.491253    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-workload-certs\") pod \"8b3cd79b-acbb-415f-bda7-43d0d4419620\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") "
Sep 05 20:02:00.491418 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.491348    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b3cd79b-acbb-415f-bda7-43d0d4419620-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8b3cd79b-acbb-415f-bda7-43d0d4419620" (UID: "8b3cd79b-acbb-415f-bda7-43d0d4419620"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:02:00.491706 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.491528    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-workload-socket\") pod \"8b3cd79b-acbb-415f-bda7-43d0d4419620\" (UID: \"8b3cd79b-acbb-415f-bda7-43d0d4419620\") "
Sep 05 20:02:00.491706 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.491567    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8b3cd79b-acbb-415f-bda7-43d0d4419620" (UID: "8b3cd79b-acbb-415f-bda7-43d0d4419620"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:02:00.491869 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.491755    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8b3cd79b-acbb-415f-bda7-43d0d4419620" (UID: "8b3cd79b-acbb-415f-bda7-43d0d4419620"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:02:00.491869 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.491843    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-istio-data" (OuterVolumeSpecName: "istio-data") pod "8b3cd79b-acbb-415f-bda7-43d0d4419620" (UID: "8b3cd79b-acbb-415f-bda7-43d0d4419620"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:02:00.491961 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.491882    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8b3cd79b-acbb-415f-bda7-43d0d4419620" (UID: "8b3cd79b-acbb-415f-bda7-43d0d4419620"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:02:00.491961 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.491945    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b3cd79b-acbb-415f-bda7-43d0d4419620-istiod-ca-cert\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:02:00.492023 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.491964    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-istio-data\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:02:00.492023 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.491977    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-credential-socket\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:02:00.492023 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.491991    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-workload-certs\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:02:00.493321 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.493292    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b3cd79b-acbb-415f-bda7-43d0d4419620-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8b3cd79b-acbb-415f-bda7-43d0d4419620" (UID: "8b3cd79b-acbb-415f-bda7-43d0d4419620"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 20:02:00.493411 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.493320    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8b3cd79b-acbb-415f-bda7-43d0d4419620" (UID: "8b3cd79b-acbb-415f-bda7-43d0d4419620"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:02:00.493524 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.493507    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b3cd79b-acbb-415f-bda7-43d0d4419620-istio-token" (OuterVolumeSpecName: "istio-token") pod "8b3cd79b-acbb-415f-bda7-43d0d4419620" (UID: "8b3cd79b-acbb-415f-bda7-43d0d4419620"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:02:00.493587 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.493522    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b3cd79b-acbb-415f-bda7-43d0d4419620-kube-api-access-7jtvg" (OuterVolumeSpecName: "kube-api-access-7jtvg") pod "8b3cd79b-acbb-415f-bda7-43d0d4419620" (UID: "8b3cd79b-acbb-415f-bda7-43d0d4419620"). InnerVolumeSpecName "kube-api-access-7jtvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:02:00.593340 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.593305    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-istio-envoy\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:02:00.593340 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.593338    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b3cd79b-acbb-415f-bda7-43d0d4419620-workload-socket\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:02:00.593582 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.593353    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jtvg\" (UniqueName: \"kubernetes.io/projected/8b3cd79b-acbb-415f-bda7-43d0d4419620-kube-api-access-7jtvg\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:02:00.593582 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.593368    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b3cd79b-acbb-415f-bda7-43d0d4419620-istio-token\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:02:00.593582 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.593382    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b3cd79b-acbb-415f-bda7-43d0d4419620-istio-podinfo\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:02:00.693579 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.693543    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"]
Sep 05 20:02:00.697917 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.697888    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-1fd18b-openshift-default-64c6b944b-gsvcs"]
Sep 05 20:02:00.963195 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.962578    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-49af4f-57676557b5-frkkl"]
Sep 05 20:02:00.964135 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.963631    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b3cd79b-acbb-415f-bda7-43d0d4419620" containerName="istio-proxy"
Sep 05 20:02:00.964135 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.963656    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b3cd79b-acbb-415f-bda7-43d0d4419620" containerName="istio-proxy"
Sep 05 20:02:00.964135 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.963893    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b3cd79b-acbb-415f-bda7-43d0d4419620" containerName="istio-proxy"
Sep 05 20:02:00.964135 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.963907    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b3cd79b-acbb-415f-bda7-43d0d4419620" containerName="istio-proxy"
Sep 05 20:02:00.975035 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.974763    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-49af4f-57676557b5-frkkl"
Sep 05 20:02:00.977104 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.977073    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-49af4f-dockercfg-f7x25\""
Sep 05 20:02:00.977805 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.977779    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-49af4f-57676557b5-frkkl"]
Sep 05 20:02:00.978707 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:00.978686    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-49af4f\""
Sep 05 20:02:01.098679 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:01.098634    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gshsc\" (UniqueName: \"kubernetes.io/projected/ed33b879-a2ca-44a4-8377-fdc18928c09b-kube-api-access-gshsc\") pod \"payload-processing-e2e-explicit-49af4f-57676557b5-frkkl\" (UID: \"ed33b879-a2ca-44a4-8377-fdc18928c09b\") " pod="openshift-ingress/payload-processing-e2e-explicit-49af4f-57676557b5-frkkl"
Sep 05 20:02:01.098871 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:01.098733    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed33b879-a2ca-44a4-8377-fdc18928c09b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-49af4f-57676557b5-frkkl\" (UID: \"ed33b879-a2ca-44a4-8377-fdc18928c09b\") " pod="openshift-ingress/payload-processing-e2e-explicit-49af4f-57676557b5-frkkl"
Sep 05 20:02:01.199741 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:01.199696    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gshsc\" (UniqueName: \"kubernetes.io/projected/ed33b879-a2ca-44a4-8377-fdc18928c09b-kube-api-access-gshsc\") pod \"payload-processing-e2e-explicit-49af4f-57676557b5-frkkl\" (UID: \"ed33b879-a2ca-44a4-8377-fdc18928c09b\") " pod="openshift-ingress/payload-processing-e2e-explicit-49af4f-57676557b5-frkkl"
Sep 05 20:02:01.199952 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:01.199790    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed33b879-a2ca-44a4-8377-fdc18928c09b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-49af4f-57676557b5-frkkl\" (UID: \"ed33b879-a2ca-44a4-8377-fdc18928c09b\") " pod="openshift-ingress/payload-processing-e2e-explicit-49af4f-57676557b5-frkkl"
Sep 05 20:02:01.200588 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:01.200563    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed33b879-a2ca-44a4-8377-fdc18928c09b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-49af4f-57676557b5-frkkl\" (UID: \"ed33b879-a2ca-44a4-8377-fdc18928c09b\") " pod="openshift-ingress/payload-processing-e2e-explicit-49af4f-57676557b5-frkkl"
Sep 05 20:02:01.208251 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:01.208222    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gshsc\" (UniqueName: \"kubernetes.io/projected/ed33b879-a2ca-44a4-8377-fdc18928c09b-kube-api-access-gshsc\") pod \"payload-processing-e2e-explicit-49af4f-57676557b5-frkkl\" (UID: \"ed33b879-a2ca-44a4-8377-fdc18928c09b\") " pod="openshift-ingress/payload-processing-e2e-explicit-49af4f-57676557b5-frkkl"
Sep 05 20:02:01.288978 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:01.288896    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-49af4f-57676557b5-frkkl"
Sep 05 20:02:01.435467 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:01.435432    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b3cd79b-acbb-415f-bda7-43d0d4419620" path="/var/lib/kubelet/pods/8b3cd79b-acbb-415f-bda7-43d0d4419620/volumes"
Sep 05 20:02:01.571658 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:01.571630    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-49af4f-57676557b5-frkkl"]
Sep 05 20:02:01.573590 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:02:01.573563    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded33b879_a2ca_44a4_8377_fdc18928c09b.slice/crio-29e81c7f56c88865e5ddae0992b8aa6e92b695ccff9c63b435680faa915e7301 WatchSource:0}: Error finding container 29e81c7f56c88865e5ddae0992b8aa6e92b695ccff9c63b435680faa915e7301: Status 404 returned error can't find the container with id 29e81c7f56c88865e5ddae0992b8aa6e92b695ccff9c63b435680faa915e7301
Sep 05 20:02:02.382469 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:02.382431    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-49af4f-57676557b5-frkkl" event={"ID":"ed33b879-a2ca-44a4-8377-fdc18928c09b","Type":"ContainerStarted","Data":"c8f52eba4ccce798af2c413ed82bd0b475958f0bedddeccf895a41ef4c4bc136"}
Sep 05 20:02:02.382682 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:02.382488    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-49af4f-57676557b5-frkkl" event={"ID":"ed33b879-a2ca-44a4-8377-fdc18928c09b","Type":"ContainerStarted","Data":"29e81c7f56c88865e5ddae0992b8aa6e92b695ccff9c63b435680faa915e7301"}
Sep 05 20:02:02.382682 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:02.382519    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-49af4f-57676557b5-frkkl"
Sep 05 20:02:02.403833 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:02.403782    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-49af4f-57676557b5-frkkl" podStartSLOduration=2.403766656 podStartE2EDuration="2.403766656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:02:02.40080715 +0000 UTC m=+1121.430439056" watchObservedRunningTime="2026-09-05 20:02:02.403766656 +0000 UTC m=+1121.433398588"
Sep 05 20:02:13.390566 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:13.390532    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-49af4f-57676557b5-frkkl"
Sep 05 20:02:36.229564 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.229522    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"]
Sep 05 20:02:36.230198 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.230177    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b3cd79b-acbb-415f-bda7-43d0d4419620" containerName="istio-proxy"
Sep 05 20:02:36.230260 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.230201    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b3cd79b-acbb-415f-bda7-43d0d4419620" containerName="istio-proxy"
Sep 05 20:02:36.235506 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.235481    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"
Sep 05 20:02:36.238674 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.238512    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-49af4f\"/\"explicit-model-49af4f-kserve-self-signed-certs\""
Sep 05 20:02:36.238674 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.238620    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-49af4f\"/\"openshift-service-ca.crt\""
Sep 05 20:02:36.238882 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.238752    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-49af4f\"/\"default-dockercfg-7xmbj\""
Sep 05 20:02:36.238939 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.238881    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-49af4f\"/\"kube-root-ca.crt\""
Sep 05 20:02:36.243502 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.243479    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"]
Sep 05 20:02:36.343987 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.343954    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfdw4\" (UniqueName: \"kubernetes.io/projected/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-kube-api-access-sfdw4\") pod \"explicit-model-49af4f-kserve-84ff59bbb5-t7tv6\" (UID: \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\") " pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"
Sep 05 20:02:36.344196 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.344000    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-tls-certs\") pod \"explicit-model-49af4f-kserve-84ff59bbb5-t7tv6\" (UID: \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\") " pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"
Sep 05 20:02:36.344196 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.344043    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-model-cache\") pod \"explicit-model-49af4f-kserve-84ff59bbb5-t7tv6\" (UID: \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\") " pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"
Sep 05 20:02:36.344196 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.344089    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-home\") pod \"explicit-model-49af4f-kserve-84ff59bbb5-t7tv6\" (UID: \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\") " pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"
Sep 05 20:02:36.344374 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.344230    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-tmp-dir\") pod \"explicit-model-49af4f-kserve-84ff59bbb5-t7tv6\" (UID: \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\") " pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"
Sep 05 20:02:36.344374 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.344279    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-dshm\") pod \"explicit-model-49af4f-kserve-84ff59bbb5-t7tv6\" (UID: \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\") " pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"
Sep 05 20:02:36.445263 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.445226    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-tmp-dir\") pod \"explicit-model-49af4f-kserve-84ff59bbb5-t7tv6\" (UID: \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\") " pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"
Sep 05 20:02:36.445263 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.445265    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-dshm\") pod \"explicit-model-49af4f-kserve-84ff59bbb5-t7tv6\" (UID: \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\") " pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"
Sep 05 20:02:36.445468 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.445389    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfdw4\" (UniqueName: \"kubernetes.io/projected/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-kube-api-access-sfdw4\") pod \"explicit-model-49af4f-kserve-84ff59bbb5-t7tv6\" (UID: \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\") " pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"
Sep 05 20:02:36.445468 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.445435    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-tls-certs\") pod \"explicit-model-49af4f-kserve-84ff59bbb5-t7tv6\" (UID: \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\") " pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"
Sep 05 20:02:36.445557 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.445478    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-model-cache\") pod \"explicit-model-49af4f-kserve-84ff59bbb5-t7tv6\" (UID: \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\") " pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"
Sep 05 20:02:36.445602 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.445565    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-home\") pod \"explicit-model-49af4f-kserve-84ff59bbb5-t7tv6\" (UID: \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\") " pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"
Sep 05 20:02:36.445650 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.445613    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-tmp-dir\") pod \"explicit-model-49af4f-kserve-84ff59bbb5-t7tv6\" (UID: \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\") " pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"
Sep 05 20:02:36.445865 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.445843    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-model-cache\") pod \"explicit-model-49af4f-kserve-84ff59bbb5-t7tv6\" (UID: \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\") " pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"
Sep 05 20:02:36.445940 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.445877    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-home\") pod \"explicit-model-49af4f-kserve-84ff59bbb5-t7tv6\" (UID: \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\") " pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"
Sep 05 20:02:36.447652 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.447631    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-dshm\") pod \"explicit-model-49af4f-kserve-84ff59bbb5-t7tv6\" (UID: \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\") " pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"
Sep 05 20:02:36.447817 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.447801    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-tls-certs\") pod \"explicit-model-49af4f-kserve-84ff59bbb5-t7tv6\" (UID: \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\") " pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"
Sep 05 20:02:36.454228 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.454209    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfdw4\" (UniqueName: \"kubernetes.io/projected/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-kube-api-access-sfdw4\") pod \"explicit-model-49af4f-kserve-84ff59bbb5-t7tv6\" (UID: \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\") " pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"
Sep 05 20:02:36.547322 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.547238    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"
Sep 05 20:02:36.820963 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:36.820937    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"]
Sep 05 20:02:36.822617 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:02:36.822592    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1dc4bff8_1fc7_4f3c_bb3a_99969447e318.slice/crio-522d5fb5fc20dedb50b841b88a9cbddfe45c7f72cac6266805e6551b709bab74 WatchSource:0}: Error finding container 522d5fb5fc20dedb50b841b88a9cbddfe45c7f72cac6266805e6551b709bab74: Status 404 returned error can't find the container with id 522d5fb5fc20dedb50b841b88a9cbddfe45c7f72cac6266805e6551b709bab74
Sep 05 20:02:37.613503 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:37.613462    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6" event={"ID":"1dc4bff8-1fc7-4f3c-bb3a-99969447e318","Type":"ContainerStarted","Data":"d98bb377bf86fc266050286c2c5b8190fa2c4c140d8b6f05e86e4d7e4ebd65fb"}
Sep 05 20:02:37.613926 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:37.613525    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6" event={"ID":"1dc4bff8-1fc7-4f3c-bb3a-99969447e318","Type":"ContainerStarted","Data":"522d5fb5fc20dedb50b841b88a9cbddfe45c7f72cac6266805e6551b709bab74"}
Sep 05 20:02:37.632173 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:37.632097    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6" podStartSLOduration=1.632081629 podStartE2EDuration="1.632081629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:02:37.631135172 +0000 UTC m=+1156.660767079" watchObservedRunningTime="2026-09-05 20:02:37.632081629 +0000 UTC m=+1156.661713534"
Sep 05 20:02:42.510207 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:42.510175    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"]
Sep 05 20:02:42.627285 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:42.624410    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-49af4f-s4cdn"]
Sep 05 20:02:42.629866 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:42.629831    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-49af4f-s4cdn"
Sep 05 20:02:42.632235 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:42.632206    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 20:02:42.632365 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:42.632206    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zx89\""
Sep 05 20:02:42.633784 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:42.633755    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-49af4f-s4cdn"]
Sep 05 20:02:42.702066 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:42.702031    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91b1886c-1172-40f9-b429-c216c381853b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-49af4f-s4cdn\" (UID: \"91b1886c-1172-40f9-b429-c216c381853b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-49af4f-s4cdn"
Sep 05 20:02:42.802800 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:42.802707    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91b1886c-1172-40f9-b429-c216c381853b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-49af4f-s4cdn\" (UID: \"91b1886c-1172-40f9-b429-c216c381853b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-49af4f-s4cdn"
Sep 05 20:02:42.803346 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:42.803327    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91b1886c-1172-40f9-b429-c216c381853b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-49af4f-s4cdn\" (UID: \"91b1886c-1172-40f9-b429-c216c381853b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-49af4f-s4cdn"
Sep 05 20:02:42.943431 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:42.943391    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-49af4f-s4cdn"
Sep 05 20:02:43.219461 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:43.219431    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-49af4f-s4cdn"]
Sep 05 20:02:43.220725 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:02:43.220687    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b1886c_1172_40f9_b429_c216c381853b.slice/crio-e06cd268027c374548c6389890ca57173040e195e728d659d704280570f41d2f WatchSource:0}: Error finding container e06cd268027c374548c6389890ca57173040e195e728d659d704280570f41d2f: Status 404 returned error can't find the container with id e06cd268027c374548c6389890ca57173040e195e728d659d704280570f41d2f
Sep 05 20:02:43.655400 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:43.655363    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-49af4f-s4cdn" event={"ID":"91b1886c-1172-40f9-b429-c216c381853b","Type":"ContainerStarted","Data":"4dd2d18f3867a4527adad353ea75e0745056142f062481849bbb0f55890f5e84"}
Sep 05 20:02:43.655764 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:43.655421    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-49af4f-s4cdn" event={"ID":"91b1886c-1172-40f9-b429-c216c381853b","Type":"ContainerStarted","Data":"e06cd268027c374548c6389890ca57173040e195e728d659d704280570f41d2f"}
Sep 05 20:02:43.686099 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:43.686046    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-49af4f-s4cdn" podStartSLOduration=1.686030695 podStartE2EDuration="1.686030695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:02:43.68293949 +0000 UTC m=+1162.712571395" watchObservedRunningTime="2026-09-05 20:02:43.686030695 +0000 UTC m=+1162.715662600"
Sep 05 20:02:44.663038 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:44.662999    2583 generic.go:356] "Generic (PLEG): container finished" podID="91b1886c-1172-40f9-b429-c216c381853b" containerID="4dd2d18f3867a4527adad353ea75e0745056142f062481849bbb0f55890f5e84" exitCode=0
Sep 05 20:02:44.663508 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:44.663081    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-49af4f-s4cdn" event={"ID":"91b1886c-1172-40f9-b429-c216c381853b","Type":"ContainerDied","Data":"4dd2d18f3867a4527adad353ea75e0745056142f062481849bbb0f55890f5e84"}
Sep 05 20:02:45.812758 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:45.812729    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-49af4f-s4cdn"
Sep 05 20:02:45.832962 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:45.832937    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91b1886c-1172-40f9-b429-c216c381853b-maas-api-service-ca\") pod \"91b1886c-1172-40f9-b429-c216c381853b\" (UID: \"91b1886c-1172-40f9-b429-c216c381853b\") "
Sep 05 20:02:45.833284 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:45.833258    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b1886c-1172-40f9-b429-c216c381853b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "91b1886c-1172-40f9-b429-c216c381853b" (UID: "91b1886c-1172-40f9-b429-c216c381853b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:02:45.934030 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:45.933950    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91b1886c-1172-40f9-b429-c216c381853b-maas-api-service-ca\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:02:46.547707 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:46.547669    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"
Sep 05 20:02:46.677037 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:46.677002    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-49af4f-s4cdn" event={"ID":"91b1886c-1172-40f9-b429-c216c381853b","Type":"ContainerDied","Data":"e06cd268027c374548c6389890ca57173040e195e728d659d704280570f41d2f"}
Sep 05 20:02:46.677251 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:46.677057    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e06cd268027c374548c6389890ca57173040e195e728d659d704280570f41d2f"
Sep 05 20:02:46.677251 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:46.677019    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-49af4f-s4cdn"
Sep 05 20:02:53.017634 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:53.017598    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"]
Sep 05 20:02:53.129811 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:53.129775    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-49af4f-57676557b5-frkkl"]
Sep 05 20:02:53.130092 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:53.130049    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-49af4f-57676557b5-frkkl" podUID="ed33b879-a2ca-44a4-8377-fdc18928c09b" containerName="payload-processing" containerID="cri-o://c8f52eba4ccce798af2c413ed82bd0b475958f0bedddeccf895a41ef4c4bc136" gracePeriod=30
Sep 05 20:02:53.327510 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:53.327457    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-49af4f-s4cdn"]
Sep 05 20:02:53.333009 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:53.332978    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-49af4f-s4cdn"]
Sep 05 20:02:53.387731 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:53.387705    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-49af4f-57676557b5-frkkl"
Sep 05 20:02:53.436589 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:53.436551    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91b1886c-1172-40f9-b429-c216c381853b" path="/var/lib/kubelet/pods/91b1886c-1172-40f9-b429-c216c381853b/volumes"
Sep 05 20:02:53.504132 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:53.504077    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gshsc\" (UniqueName: \"kubernetes.io/projected/ed33b879-a2ca-44a4-8377-fdc18928c09b-kube-api-access-gshsc\") pod \"ed33b879-a2ca-44a4-8377-fdc18928c09b\" (UID: \"ed33b879-a2ca-44a4-8377-fdc18928c09b\") "
Sep 05 20:02:53.504318 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:53.504159    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed33b879-a2ca-44a4-8377-fdc18928c09b-plugins-config-volume\") pod \"ed33b879-a2ca-44a4-8377-fdc18928c09b\" (UID: \"ed33b879-a2ca-44a4-8377-fdc18928c09b\") "
Sep 05 20:02:53.504541 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:53.504507    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed33b879-a2ca-44a4-8377-fdc18928c09b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ed33b879-a2ca-44a4-8377-fdc18928c09b" (UID: "ed33b879-a2ca-44a4-8377-fdc18928c09b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:02:53.506308 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:53.506282    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed33b879-a2ca-44a4-8377-fdc18928c09b-kube-api-access-gshsc" (OuterVolumeSpecName: "kube-api-access-gshsc") pod "ed33b879-a2ca-44a4-8377-fdc18928c09b" (UID: "ed33b879-a2ca-44a4-8377-fdc18928c09b"). InnerVolumeSpecName "kube-api-access-gshsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:02:53.604925 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:53.604890    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gshsc\" (UniqueName: \"kubernetes.io/projected/ed33b879-a2ca-44a4-8377-fdc18928c09b-kube-api-access-gshsc\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:02:53.604925 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:53.604926    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed33b879-a2ca-44a4-8377-fdc18928c09b-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:02:53.722798 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:53.722760    2583 generic.go:356] "Generic (PLEG): container finished" podID="ed33b879-a2ca-44a4-8377-fdc18928c09b" containerID="c8f52eba4ccce798af2c413ed82bd0b475958f0bedddeccf895a41ef4c4bc136" exitCode=0
Sep 05 20:02:53.722986 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:53.722805    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-49af4f-57676557b5-frkkl"
Sep 05 20:02:53.722986 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:53.722848    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-49af4f-57676557b5-frkkl" event={"ID":"ed33b879-a2ca-44a4-8377-fdc18928c09b","Type":"ContainerDied","Data":"c8f52eba4ccce798af2c413ed82bd0b475958f0bedddeccf895a41ef4c4bc136"}
Sep 05 20:02:53.722986 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:53.722904    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-49af4f-57676557b5-frkkl" event={"ID":"ed33b879-a2ca-44a4-8377-fdc18928c09b","Type":"ContainerDied","Data":"29e81c7f56c88865e5ddae0992b8aa6e92b695ccff9c63b435680faa915e7301"}
Sep 05 20:02:53.722986 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:53.722935    2583 scope.go:117] "RemoveContainer" containerID="c8f52eba4ccce798af2c413ed82bd0b475958f0bedddeccf895a41ef4c4bc136"
Sep 05 20:02:53.733161 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:53.733143    2583 scope.go:117] "RemoveContainer" containerID="c8f52eba4ccce798af2c413ed82bd0b475958f0bedddeccf895a41ef4c4bc136"
Sep 05 20:02:53.733413 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:02:53.733394    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f52eba4ccce798af2c413ed82bd0b475958f0bedddeccf895a41ef4c4bc136\": container with ID starting with c8f52eba4ccce798af2c413ed82bd0b475958f0bedddeccf895a41ef4c4bc136 not found: ID does not exist" containerID="c8f52eba4ccce798af2c413ed82bd0b475958f0bedddeccf895a41ef4c4bc136"
Sep 05 20:02:53.733464 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:53.733422    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f52eba4ccce798af2c413ed82bd0b475958f0bedddeccf895a41ef4c4bc136"} err="failed to get container status \"c8f52eba4ccce798af2c413ed82bd0b475958f0bedddeccf895a41ef4c4bc136\": rpc error: code = NotFound desc = could not find container \"c8f52eba4ccce798af2c413ed82bd0b475958f0bedddeccf895a41ef4c4bc136\": container with ID starting with c8f52eba4ccce798af2c413ed82bd0b475958f0bedddeccf895a41ef4c4bc136 not found: ID does not exist"
Sep 05 20:02:53.748213 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:53.748188    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-49af4f-57676557b5-frkkl"]
Sep 05 20:02:53.754487 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:53.754467    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-49af4f-57676557b5-frkkl"]
Sep 05 20:02:55.435028 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:55.434992    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed33b879-a2ca-44a4-8377-fdc18928c09b" path="/var/lib/kubelet/pods/ed33b879-a2ca-44a4-8377-fdc18928c09b/volumes"
Sep 05 20:02:57.530203 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:57.530141    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6" podUID="1dc4bff8-1fc7-4f3c-bb3a-99969447e318" containerName="main" containerID="cri-o://d98bb377bf86fc266050286c2c5b8190fa2c4c140d8b6f05e86e4d7e4ebd65fb" gracePeriod=45
Sep 05 20:02:57.750789 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:57.750758    2583 generic.go:356] "Generic (PLEG): container finished" podID="1dc4bff8-1fc7-4f3c-bb3a-99969447e318" containerID="d98bb377bf86fc266050286c2c5b8190fa2c4c140d8b6f05e86e4d7e4ebd65fb" exitCode=0
Sep 05 20:02:57.750944 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:57.750829    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6" event={"ID":"1dc4bff8-1fc7-4f3c-bb3a-99969447e318","Type":"ContainerDied","Data":"d98bb377bf86fc266050286c2c5b8190fa2c4c140d8b6f05e86e4d7e4ebd65fb"}
Sep 05 20:02:57.800017 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:57.799992    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"
Sep 05 20:02:57.846965 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:57.846925    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-dshm\") pod \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\" (UID: \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\") "
Sep 05 20:02:57.847163 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:57.847033    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-tls-certs\") pod \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\" (UID: \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\") "
Sep 05 20:02:57.847163 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:57.847068    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-model-cache\") pod \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\" (UID: \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\") "
Sep 05 20:02:57.847301 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:57.847177    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfdw4\" (UniqueName: \"kubernetes.io/projected/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-kube-api-access-sfdw4\") pod \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\" (UID: \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\") "
Sep 05 20:02:57.847301 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:57.847269    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-home\") pod \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\" (UID: \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\") "
Sep 05 20:02:57.847412 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:57.847311    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-tmp-dir\") pod \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\" (UID: \"1dc4bff8-1fc7-4f3c-bb3a-99969447e318\") "
Sep 05 20:02:57.847412 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:57.847344    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-model-cache" (OuterVolumeSpecName: "model-cache") pod "1dc4bff8-1fc7-4f3c-bb3a-99969447e318" (UID: "1dc4bff8-1fc7-4f3c-bb3a-99969447e318"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:02:57.847552 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:57.847532    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-home" (OuterVolumeSpecName: "home") pod "1dc4bff8-1fc7-4f3c-bb3a-99969447e318" (UID: "1dc4bff8-1fc7-4f3c-bb3a-99969447e318"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:02:57.847704 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:57.847680    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1dc4bff8-1fc7-4f3c-bb3a-99969447e318" (UID: "1dc4bff8-1fc7-4f3c-bb3a-99969447e318"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:02:57.847704 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:57.847670    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-home\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:02:57.847884 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:57.847726    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-model-cache\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:02:57.849466 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:57.849438    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1dc4bff8-1fc7-4f3c-bb3a-99969447e318" (UID: "1dc4bff8-1fc7-4f3c-bb3a-99969447e318"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:02:57.849645 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:57.849623    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-dshm" (OuterVolumeSpecName: "dshm") pod "1dc4bff8-1fc7-4f3c-bb3a-99969447e318" (UID: "1dc4bff8-1fc7-4f3c-bb3a-99969447e318"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:02:57.849714 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:57.849644    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-kube-api-access-sfdw4" (OuterVolumeSpecName: "kube-api-access-sfdw4") pod "1dc4bff8-1fc7-4f3c-bb3a-99969447e318" (UID: "1dc4bff8-1fc7-4f3c-bb3a-99969447e318"). InnerVolumeSpecName "kube-api-access-sfdw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:02:57.949069 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:57.949033    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-tls-certs\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:02:57.949069 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:57.949064    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfdw4\" (UniqueName: \"kubernetes.io/projected/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-kube-api-access-sfdw4\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:02:57.949069 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:57.949076    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-tmp-dir\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:02:57.949313 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:57.949085    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dc4bff8-1fc7-4f3c-bb3a-99969447e318-dshm\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:02:58.758054 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:58.758022    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"
Sep 05 20:02:58.758473 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:58.758026    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6" event={"ID":"1dc4bff8-1fc7-4f3c-bb3a-99969447e318","Type":"ContainerDied","Data":"522d5fb5fc20dedb50b841b88a9cbddfe45c7f72cac6266805e6551b709bab74"}
Sep 05 20:02:58.758473 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:58.758167    2583 scope.go:117] "RemoveContainer" containerID="d98bb377bf86fc266050286c2c5b8190fa2c4c140d8b6f05e86e4d7e4ebd65fb"
Sep 05 20:02:58.779772 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:58.779746    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"]
Sep 05 20:02:58.783791 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:58.783765    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-49af4f/explicit-model-49af4f-kserve-84ff59bbb5-t7tv6"]
Sep 05 20:02:59.435780 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:02:59.435744    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dc4bff8-1fc7-4f3c-bb3a-99969447e318" path="/var/lib/kubelet/pods/1dc4bff8-1fc7-4f3c-bb3a-99969447e318/volumes"
Sep 05 20:03:03.721676 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:03.721629    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-588081-29810640-5fvzr"]
Sep 05 20:03:03.726151 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:03.726101    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-588081-29810640-5fvzr"]
Sep 05 20:03:03.748051 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:03.748016    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm"]
Sep 05 20:03:03.748641 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:03.748581    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm" podUID="3d96c4ef-3daa-42ae-af20-d4640421f102" containerName="payload-processing" containerID="cri-o://9ea1e8d50270171b8db446041e7279e01e3fdc25ce430bae178571478e495bc0" gracePeriod=30
Sep 05 20:03:03.755083 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:03.755050    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs"]
Sep 05 20:03:03.755339 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:03.755312    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs" podUID="00416952-66bf-46d7-84bb-617343832ad5" containerName="payload-pre-processing" containerID="cri-o://56a79ffff1a899611ae78f8b3ffaf4e505d5af55ad8f7846649e52c47ca1c682" gracePeriod=30
Sep 05 20:03:03.945048 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:03.944987    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs" podUID="00416952-66bf-46d7-84bb-617343832ad5" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.52:9004: connect: connection refused"
Sep 05 20:03:04.034974 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.034943    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm"
Sep 05 20:03:04.046429 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.046405    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs"
Sep 05 20:03:04.111673 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.111636    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00416952-66bf-46d7-84bb-617343832ad5-plugins-config-volume\") pod \"00416952-66bf-46d7-84bb-617343832ad5\" (UID: \"00416952-66bf-46d7-84bb-617343832ad5\") "
Sep 05 20:03:04.111869 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.111698    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6pt6\" (UniqueName: \"kubernetes.io/projected/00416952-66bf-46d7-84bb-617343832ad5-kube-api-access-l6pt6\") pod \"00416952-66bf-46d7-84bb-617343832ad5\" (UID: \"00416952-66bf-46d7-84bb-617343832ad5\") "
Sep 05 20:03:04.111869 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.111733    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h956b\" (UniqueName: \"kubernetes.io/projected/3d96c4ef-3daa-42ae-af20-d4640421f102-kube-api-access-h956b\") pod \"3d96c4ef-3daa-42ae-af20-d4640421f102\" (UID: \"3d96c4ef-3daa-42ae-af20-d4640421f102\") "
Sep 05 20:03:04.111869 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.111763    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d96c4ef-3daa-42ae-af20-d4640421f102-plugins-config-volume\") pod \"3d96c4ef-3daa-42ae-af20-d4640421f102\" (UID: \"3d96c4ef-3daa-42ae-af20-d4640421f102\") "
Sep 05 20:03:04.112240 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.112200    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00416952-66bf-46d7-84bb-617343832ad5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "00416952-66bf-46d7-84bb-617343832ad5" (UID: "00416952-66bf-46d7-84bb-617343832ad5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:03:04.112381 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.112241    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d96c4ef-3daa-42ae-af20-d4640421f102-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d96c4ef-3daa-42ae-af20-d4640421f102" (UID: "3d96c4ef-3daa-42ae-af20-d4640421f102"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:03:04.113883 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.113861    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00416952-66bf-46d7-84bb-617343832ad5-kube-api-access-l6pt6" (OuterVolumeSpecName: "kube-api-access-l6pt6") pod "00416952-66bf-46d7-84bb-617343832ad5" (UID: "00416952-66bf-46d7-84bb-617343832ad5"). InnerVolumeSpecName "kube-api-access-l6pt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:03:04.113967 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.113899    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d96c4ef-3daa-42ae-af20-d4640421f102-kube-api-access-h956b" (OuterVolumeSpecName: "kube-api-access-h956b") pod "3d96c4ef-3daa-42ae-af20-d4640421f102" (UID: "3d96c4ef-3daa-42ae-af20-d4640421f102"). InnerVolumeSpecName "kube-api-access-h956b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:03:04.212682 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.212643    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h956b\" (UniqueName: \"kubernetes.io/projected/3d96c4ef-3daa-42ae-af20-d4640421f102-kube-api-access-h956b\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:03:04.212682 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.212673    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d96c4ef-3daa-42ae-af20-d4640421f102-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:03:04.212682 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.212682    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00416952-66bf-46d7-84bb-617343832ad5-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:03:04.212682 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.212693    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6pt6\" (UniqueName: \"kubernetes.io/projected/00416952-66bf-46d7-84bb-617343832ad5-kube-api-access-l6pt6\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:03:04.803217 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.803181    2583 generic.go:356] "Generic (PLEG): container finished" podID="00416952-66bf-46d7-84bb-617343832ad5" containerID="56a79ffff1a899611ae78f8b3ffaf4e505d5af55ad8f7846649e52c47ca1c682" exitCode=0
Sep 05 20:03:04.803684 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.803240    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs"
Sep 05 20:03:04.803684 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.803272    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs" event={"ID":"00416952-66bf-46d7-84bb-617343832ad5","Type":"ContainerDied","Data":"56a79ffff1a899611ae78f8b3ffaf4e505d5af55ad8f7846649e52c47ca1c682"}
Sep 05 20:03:04.803684 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.803333    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs" event={"ID":"00416952-66bf-46d7-84bb-617343832ad5","Type":"ContainerDied","Data":"6f1fbb177512b7961a13e9c63bd230008354336fff9e34a32776f9008e5be63b"}
Sep 05 20:03:04.803684 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.803369    2583 scope.go:117] "RemoveContainer" containerID="56a79ffff1a899611ae78f8b3ffaf4e505d5af55ad8f7846649e52c47ca1c682"
Sep 05 20:03:04.805229 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.805059    2583 generic.go:356] "Generic (PLEG): container finished" podID="3d96c4ef-3daa-42ae-af20-d4640421f102" containerID="9ea1e8d50270171b8db446041e7279e01e3fdc25ce430bae178571478e495bc0" exitCode=0
Sep 05 20:03:04.805229 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.805136    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm"
Sep 05 20:03:04.805229 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.805093    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm" event={"ID":"3d96c4ef-3daa-42ae-af20-d4640421f102","Type":"ContainerDied","Data":"9ea1e8d50270171b8db446041e7279e01e3fdc25ce430bae178571478e495bc0"}
Sep 05 20:03:04.805482 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.805243    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm" event={"ID":"3d96c4ef-3daa-42ae-af20-d4640421f102","Type":"ContainerDied","Data":"b06c8173ef8deba935948805f0c9dfd039219ccd20edcbf0e1e083dc719cc583"}
Sep 05 20:03:04.815549 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.815530    2583 scope.go:117] "RemoveContainer" containerID="56a79ffff1a899611ae78f8b3ffaf4e505d5af55ad8f7846649e52c47ca1c682"
Sep 05 20:03:04.816567 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:03:04.816527    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56a79ffff1a899611ae78f8b3ffaf4e505d5af55ad8f7846649e52c47ca1c682\": container with ID starting with 56a79ffff1a899611ae78f8b3ffaf4e505d5af55ad8f7846649e52c47ca1c682 not found: ID does not exist" containerID="56a79ffff1a899611ae78f8b3ffaf4e505d5af55ad8f7846649e52c47ca1c682"
Sep 05 20:03:04.816708 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.816565    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56a79ffff1a899611ae78f8b3ffaf4e505d5af55ad8f7846649e52c47ca1c682"} err="failed to get container status \"56a79ffff1a899611ae78f8b3ffaf4e505d5af55ad8f7846649e52c47ca1c682\": rpc error: code = NotFound desc = could not find container \"56a79ffff1a899611ae78f8b3ffaf4e505d5af55ad8f7846649e52c47ca1c682\": container with ID starting with 56a79ffff1a899611ae78f8b3ffaf4e505d5af55ad8f7846649e52c47ca1c682 not found: ID does not exist"
Sep 05 20:03:04.816708 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.816588    2583 scope.go:117] "RemoveContainer" containerID="9ea1e8d50270171b8db446041e7279e01e3fdc25ce430bae178571478e495bc0"
Sep 05 20:03:04.826907 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.826886    2583 scope.go:117] "RemoveContainer" containerID="9ea1e8d50270171b8db446041e7279e01e3fdc25ce430bae178571478e495bc0"
Sep 05 20:03:04.827222 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:03:04.827203    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea1e8d50270171b8db446041e7279e01e3fdc25ce430bae178571478e495bc0\": container with ID starting with 9ea1e8d50270171b8db446041e7279e01e3fdc25ce430bae178571478e495bc0 not found: ID does not exist" containerID="9ea1e8d50270171b8db446041e7279e01e3fdc25ce430bae178571478e495bc0"
Sep 05 20:03:04.827284 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.827228    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea1e8d50270171b8db446041e7279e01e3fdc25ce430bae178571478e495bc0"} err="failed to get container status \"9ea1e8d50270171b8db446041e7279e01e3fdc25ce430bae178571478e495bc0\": rpc error: code = NotFound desc = could not find container \"9ea1e8d50270171b8db446041e7279e01e3fdc25ce430bae178571478e495bc0\": container with ID starting with 9ea1e8d50270171b8db446041e7279e01e3fdc25ce430bae178571478e495bc0 not found: ID does not exist"
Sep 05 20:03:04.834889 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.834861    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm"]
Sep 05 20:03:04.838689 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.838667    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm"]
Sep 05 20:03:04.847559 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.847524    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs"]
Sep 05 20:03:04.851498 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.851467    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-588081-6585db478b-r4vjs"]
Sep 05 20:03:04.945531 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:04.945486    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-588081-59665c9769-gn8pm" podUID="3d96c4ef-3daa-42ae-af20-d4640421f102" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.53:9004: i/o timeout"
Sep 05 20:03:05.435207 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:05.435171    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00416952-66bf-46d7-84bb-617343832ad5" path="/var/lib/kubelet/pods/00416952-66bf-46d7-84bb-617343832ad5/volumes"
Sep 05 20:03:05.435550 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:05.435538    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d96c4ef-3daa-42ae-af20-d4640421f102" path="/var/lib/kubelet/pods/3d96c4ef-3daa-42ae-af20-d4640421f102/volumes"
Sep 05 20:03:05.435777 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:05.435766    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="597eb445-16dd-408d-8e84-1efc06db936c" path="/var/lib/kubelet/pods/597eb445-16dd-408d-8e84-1efc06db936c/volumes"
Sep 05 20:03:08.038288 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.038242    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq" podUID="8687da3a-7a11-4a9f-9f72-95dc214f413f" containerName="main" containerID="cri-o://2989a97ab2f25fbe86a91b87e6b80c173fd615619993b2d62456c460544b6e8f" gracePeriod=45
Sep 05 20:03:08.295156 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.295057    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:03:08.456246 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.456207    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8687da3a-7a11-4a9f-9f72-95dc214f413f-model-cache\") pod \"8687da3a-7a11-4a9f-9f72-95dc214f413f\" (UID: \"8687da3a-7a11-4a9f-9f72-95dc214f413f\") "
Sep 05 20:03:08.456451 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.456255    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pw7q\" (UniqueName: \"kubernetes.io/projected/8687da3a-7a11-4a9f-9f72-95dc214f413f-kube-api-access-6pw7q\") pod \"8687da3a-7a11-4a9f-9f72-95dc214f413f\" (UID: \"8687da3a-7a11-4a9f-9f72-95dc214f413f\") "
Sep 05 20:03:08.456451 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.456278    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8687da3a-7a11-4a9f-9f72-95dc214f413f-home\") pod \"8687da3a-7a11-4a9f-9f72-95dc214f413f\" (UID: \"8687da3a-7a11-4a9f-9f72-95dc214f413f\") "
Sep 05 20:03:08.456451 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.456341    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8687da3a-7a11-4a9f-9f72-95dc214f413f-tmp-dir\") pod \"8687da3a-7a11-4a9f-9f72-95dc214f413f\" (UID: \"8687da3a-7a11-4a9f-9f72-95dc214f413f\") "
Sep 05 20:03:08.456451 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.456407    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8687da3a-7a11-4a9f-9f72-95dc214f413f-tls-certs\") pod \"8687da3a-7a11-4a9f-9f72-95dc214f413f\" (UID: \"8687da3a-7a11-4a9f-9f72-95dc214f413f\") "
Sep 05 20:03:08.456681 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.456475    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8687da3a-7a11-4a9f-9f72-95dc214f413f-dshm\") pod \"8687da3a-7a11-4a9f-9f72-95dc214f413f\" (UID: \"8687da3a-7a11-4a9f-9f72-95dc214f413f\") "
Sep 05 20:03:08.456681 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.456502    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8687da3a-7a11-4a9f-9f72-95dc214f413f-model-cache" (OuterVolumeSpecName: "model-cache") pod "8687da3a-7a11-4a9f-9f72-95dc214f413f" (UID: "8687da3a-7a11-4a9f-9f72-95dc214f413f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:03:08.456681 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.456602    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8687da3a-7a11-4a9f-9f72-95dc214f413f-home" (OuterVolumeSpecName: "home") pod "8687da3a-7a11-4a9f-9f72-95dc214f413f" (UID: "8687da3a-7a11-4a9f-9f72-95dc214f413f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:03:08.456787 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.456692    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8687da3a-7a11-4a9f-9f72-95dc214f413f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8687da3a-7a11-4a9f-9f72-95dc214f413f" (UID: "8687da3a-7a11-4a9f-9f72-95dc214f413f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:03:08.456851 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.456833    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8687da3a-7a11-4a9f-9f72-95dc214f413f-model-cache\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:03:08.456916 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.456856    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8687da3a-7a11-4a9f-9f72-95dc214f413f-home\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:03:08.456916 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.456866    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8687da3a-7a11-4a9f-9f72-95dc214f413f-tmp-dir\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:03:08.458631 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.458604    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8687da3a-7a11-4a9f-9f72-95dc214f413f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8687da3a-7a11-4a9f-9f72-95dc214f413f" (UID: "8687da3a-7a11-4a9f-9f72-95dc214f413f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:03:08.458631 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.458622    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8687da3a-7a11-4a9f-9f72-95dc214f413f-kube-api-access-6pw7q" (OuterVolumeSpecName: "kube-api-access-6pw7q") pod "8687da3a-7a11-4a9f-9f72-95dc214f413f" (UID: "8687da3a-7a11-4a9f-9f72-95dc214f413f"). InnerVolumeSpecName "kube-api-access-6pw7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:03:08.458748 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.458689    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8687da3a-7a11-4a9f-9f72-95dc214f413f-dshm" (OuterVolumeSpecName: "dshm") pod "8687da3a-7a11-4a9f-9f72-95dc214f413f" (UID: "8687da3a-7a11-4a9f-9f72-95dc214f413f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:03:08.557975 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.557877    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8687da3a-7a11-4a9f-9f72-95dc214f413f-dshm\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:03:08.557975 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.557911    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pw7q\" (UniqueName: \"kubernetes.io/projected/8687da3a-7a11-4a9f-9f72-95dc214f413f-kube-api-access-6pw7q\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:03:08.557975 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.557925    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8687da3a-7a11-4a9f-9f72-95dc214f413f-tls-certs\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:03:08.834032 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.833998    2583 generic.go:356] "Generic (PLEG): container finished" podID="8687da3a-7a11-4a9f-9f72-95dc214f413f" containerID="2989a97ab2f25fbe86a91b87e6b80c173fd615619993b2d62456c460544b6e8f" exitCode=0
Sep 05 20:03:08.834242 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.834051    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"
Sep 05 20:03:08.834242 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.834083    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq" event={"ID":"8687da3a-7a11-4a9f-9f72-95dc214f413f","Type":"ContainerDied","Data":"2989a97ab2f25fbe86a91b87e6b80c173fd615619993b2d62456c460544b6e8f"}
Sep 05 20:03:08.834242 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.834142    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq" event={"ID":"8687da3a-7a11-4a9f-9f72-95dc214f413f","Type":"ContainerDied","Data":"83331a919e2bbc0327004c9d33841566816c94c539d54e9f57ddd4541440750c"}
Sep 05 20:03:08.834242 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.834168    2583 scope.go:117] "RemoveContainer" containerID="2989a97ab2f25fbe86a91b87e6b80c173fd615619993b2d62456c460544b6e8f"
Sep 05 20:03:08.844276 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.844255    2583 scope.go:117] "RemoveContainer" containerID="2989a97ab2f25fbe86a91b87e6b80c173fd615619993b2d62456c460544b6e8f"
Sep 05 20:03:08.844553 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:03:08.844536    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2989a97ab2f25fbe86a91b87e6b80c173fd615619993b2d62456c460544b6e8f\": container with ID starting with 2989a97ab2f25fbe86a91b87e6b80c173fd615619993b2d62456c460544b6e8f not found: ID does not exist" containerID="2989a97ab2f25fbe86a91b87e6b80c173fd615619993b2d62456c460544b6e8f"
Sep 05 20:03:08.844615 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.844566    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2989a97ab2f25fbe86a91b87e6b80c173fd615619993b2d62456c460544b6e8f"} err="failed to get container status \"2989a97ab2f25fbe86a91b87e6b80c173fd615619993b2d62456c460544b6e8f\": rpc error: code = NotFound desc = could not find container \"2989a97ab2f25fbe86a91b87e6b80c173fd615619993b2d62456c460544b6e8f\": container with ID starting with 2989a97ab2f25fbe86a91b87e6b80c173fd615619993b2d62456c460544b6e8f not found: ID does not exist"
Sep 05 20:03:08.864262 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.864202    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"]
Sep 05 20:03:08.868171 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:08.868139    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1dc7b1/test-model-1dc7b1-kserve-759fbc96b9-6cnwq"]
Sep 05 20:03:09.435483 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:09.435450    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8687da3a-7a11-4a9f-9f72-95dc214f413f" path="/var/lib/kubelet/pods/8687da3a-7a11-4a9f-9f72-95dc214f413f/volumes"
Sep 05 20:03:10.377434 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:10.377404    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"]
Sep 05 20:03:10.377678 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:10.377637    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66" podUID="56770103-b7ef-4c4c-8868-03e81ec44ad5" containerName="istio-proxy" containerID="cri-o://9066fedb0e16a843eabd235b7028f4bce73a344d2b8d8e4f664bfffc407fc26c" gracePeriod=30
Sep 05 20:03:15.637989 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.637964    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 20:03:15.829169 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.829131    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56770103-b7ef-4c4c-8868-03e81ec44ad5-istio-podinfo\") pod \"56770103-b7ef-4c4c-8868-03e81ec44ad5\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") "
Sep 05 20:03:15.829357 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.829182    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-workload-certs\") pod \"56770103-b7ef-4c4c-8868-03e81ec44ad5\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") "
Sep 05 20:03:15.829357 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.829204    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-workload-socket\") pod \"56770103-b7ef-4c4c-8868-03e81ec44ad5\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") "
Sep 05 20:03:15.829357 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.829238    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56770103-b7ef-4c4c-8868-03e81ec44ad5-istio-token\") pod \"56770103-b7ef-4c4c-8868-03e81ec44ad5\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") "
Sep 05 20:03:15.829357 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.829274    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzczt\" (UniqueName: \"kubernetes.io/projected/56770103-b7ef-4c4c-8868-03e81ec44ad5-kube-api-access-vzczt\") pod \"56770103-b7ef-4c4c-8868-03e81ec44ad5\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") "
Sep 05 20:03:15.829357 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.829306    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-istio-envoy\") pod \"56770103-b7ef-4c4c-8868-03e81ec44ad5\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") "
Sep 05 20:03:15.829637 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.829388    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-credential-socket\") pod \"56770103-b7ef-4c4c-8868-03e81ec44ad5\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") "
Sep 05 20:03:15.829637 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.829412    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-istio-data\") pod \"56770103-b7ef-4c4c-8868-03e81ec44ad5\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") "
Sep 05 20:03:15.829637 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.829441    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56770103-b7ef-4c4c-8868-03e81ec44ad5-istiod-ca-cert\") pod \"56770103-b7ef-4c4c-8868-03e81ec44ad5\" (UID: \"56770103-b7ef-4c4c-8868-03e81ec44ad5\") "
Sep 05 20:03:15.829637 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.829478    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "56770103-b7ef-4c4c-8868-03e81ec44ad5" (UID: "56770103-b7ef-4c4c-8868-03e81ec44ad5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:03:15.829637 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.829499    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "56770103-b7ef-4c4c-8868-03e81ec44ad5" (UID: "56770103-b7ef-4c4c-8868-03e81ec44ad5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:03:15.829637 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.829633    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "56770103-b7ef-4c4c-8868-03e81ec44ad5" (UID: "56770103-b7ef-4c4c-8868-03e81ec44ad5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:03:15.829939 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.829674    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-istio-data" (OuterVolumeSpecName: "istio-data") pod "56770103-b7ef-4c4c-8868-03e81ec44ad5" (UID: "56770103-b7ef-4c4c-8868-03e81ec44ad5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:03:15.829939 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.829774    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-credential-socket\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:03:15.829939 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.829793    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-istio-data\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:03:15.829939 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.829806    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-workload-certs\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:03:15.829939 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.829818    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-workload-socket\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:03:15.829939 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.829875    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56770103-b7ef-4c4c-8868-03e81ec44ad5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "56770103-b7ef-4c4c-8868-03e81ec44ad5" (UID: "56770103-b7ef-4c4c-8868-03e81ec44ad5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:03:15.831601 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.831573    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56770103-b7ef-4c4c-8868-03e81ec44ad5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "56770103-b7ef-4c4c-8868-03e81ec44ad5" (UID: "56770103-b7ef-4c4c-8868-03e81ec44ad5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 20:03:15.831601 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.831580    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56770103-b7ef-4c4c-8868-03e81ec44ad5-istio-token" (OuterVolumeSpecName: "istio-token") pod "56770103-b7ef-4c4c-8868-03e81ec44ad5" (UID: "56770103-b7ef-4c4c-8868-03e81ec44ad5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:03:15.831735 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.831598    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56770103-b7ef-4c4c-8868-03e81ec44ad5-kube-api-access-vzczt" (OuterVolumeSpecName: "kube-api-access-vzczt") pod "56770103-b7ef-4c4c-8868-03e81ec44ad5" (UID: "56770103-b7ef-4c4c-8868-03e81ec44ad5"). InnerVolumeSpecName "kube-api-access-vzczt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:03:15.831735 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.831583    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "56770103-b7ef-4c4c-8868-03e81ec44ad5" (UID: "56770103-b7ef-4c4c-8868-03e81ec44ad5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:03:15.885291 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.885257    2583 generic.go:356] "Generic (PLEG): container finished" podID="56770103-b7ef-4c4c-8868-03e81ec44ad5" containerID="9066fedb0e16a843eabd235b7028f4bce73a344d2b8d8e4f664bfffc407fc26c" exitCode=0
Sep 05 20:03:15.885498 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.885306    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"
Sep 05 20:03:15.885498 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.885344    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66" event={"ID":"56770103-b7ef-4c4c-8868-03e81ec44ad5","Type":"ContainerDied","Data":"9066fedb0e16a843eabd235b7028f4bce73a344d2b8d8e4f664bfffc407fc26c"}
Sep 05 20:03:15.885498 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.885397    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66" event={"ID":"56770103-b7ef-4c4c-8868-03e81ec44ad5","Type":"ContainerDied","Data":"2107eb92739f59692854e40d141b991267098c68033376bd2e9e925eb037f6c8"}
Sep 05 20:03:15.885498 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.885422    2583 scope.go:117] "RemoveContainer" containerID="9066fedb0e16a843eabd235b7028f4bce73a344d2b8d8e4f664bfffc407fc26c"
Sep 05 20:03:15.895051 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.895030    2583 scope.go:117] "RemoveContainer" containerID="00464ff44930788a5c94267fc00754c3a4d59947c73d329267cd032ecdba962d"
Sep 05 20:03:15.903497 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.903455    2583 scope.go:117] "RemoveContainer" containerID="9066fedb0e16a843eabd235b7028f4bce73a344d2b8d8e4f664bfffc407fc26c"
Sep 05 20:03:15.903875 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:03:15.903835    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9066fedb0e16a843eabd235b7028f4bce73a344d2b8d8e4f664bfffc407fc26c\": container with ID starting with 9066fedb0e16a843eabd235b7028f4bce73a344d2b8d8e4f664bfffc407fc26c not found: ID does not exist" containerID="9066fedb0e16a843eabd235b7028f4bce73a344d2b8d8e4f664bfffc407fc26c"
Sep 05 20:03:15.903993 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.903885    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9066fedb0e16a843eabd235b7028f4bce73a344d2b8d8e4f664bfffc407fc26c"} err="failed to get container status \"9066fedb0e16a843eabd235b7028f4bce73a344d2b8d8e4f664bfffc407fc26c\": rpc error: code = NotFound desc = could not find container \"9066fedb0e16a843eabd235b7028f4bce73a344d2b8d8e4f664bfffc407fc26c\": container with ID starting with 9066fedb0e16a843eabd235b7028f4bce73a344d2b8d8e4f664bfffc407fc26c not found: ID does not exist"
Sep 05 20:03:15.903993 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.903913    2583 scope.go:117] "RemoveContainer" containerID="00464ff44930788a5c94267fc00754c3a4d59947c73d329267cd032ecdba962d"
Sep 05 20:03:15.904272 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:03:15.904242    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00464ff44930788a5c94267fc00754c3a4d59947c73d329267cd032ecdba962d\": container with ID starting with 00464ff44930788a5c94267fc00754c3a4d59947c73d329267cd032ecdba962d not found: ID does not exist" containerID="00464ff44930788a5c94267fc00754c3a4d59947c73d329267cd032ecdba962d"
Sep 05 20:03:15.904386 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.904279    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00464ff44930788a5c94267fc00754c3a4d59947c73d329267cd032ecdba962d"} err="failed to get container status \"00464ff44930788a5c94267fc00754c3a4d59947c73d329267cd032ecdba962d\": rpc error: code = NotFound desc = could not find container \"00464ff44930788a5c94267fc00754c3a4d59947c73d329267cd032ecdba962d\": container with ID starting with 00464ff44930788a5c94267fc00754c3a4d59947c73d329267cd032ecdba962d not found: ID does not exist"
Sep 05 20:03:15.907524 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.907507    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"]
Sep 05 20:03:15.912245 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.912226    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-588081-openshift-default-648bd9598-kmp66"]
Sep 05 20:03:15.931003 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.930979    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56770103-b7ef-4c4c-8868-03e81ec44ad5-istiod-ca-cert\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:03:15.931003 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.931002    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56770103-b7ef-4c4c-8868-03e81ec44ad5-istio-podinfo\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:03:15.931154 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.931014    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56770103-b7ef-4c4c-8868-03e81ec44ad5-istio-token\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:03:15.931154 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.931027    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzczt\" (UniqueName: \"kubernetes.io/projected/56770103-b7ef-4c4c-8868-03e81ec44ad5-kube-api-access-vzczt\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:03:15.931154 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:15.931040    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56770103-b7ef-4c4c-8868-03e81ec44ad5-istio-envoy\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:03:17.435203 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:17.435169    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56770103-b7ef-4c4c-8868-03e81ec44ad5" path="/var/lib/kubelet/pods/56770103-b7ef-4c4c-8868-03e81ec44ad5/volumes"
Sep 05 20:03:19.500872 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.500835    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk"]
Sep 05 20:03:19.501361 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.501342    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00416952-66bf-46d7-84bb-617343832ad5" containerName="payload-pre-processing"
Sep 05 20:03:19.501361 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.501362    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="00416952-66bf-46d7-84bb-617343832ad5" containerName="payload-pre-processing"
Sep 05 20:03:19.501466 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.501370    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d96c4ef-3daa-42ae-af20-d4640421f102" containerName="payload-processing"
Sep 05 20:03:19.501466 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.501377    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d96c4ef-3daa-42ae-af20-d4640421f102" containerName="payload-processing"
Sep 05 20:03:19.501466 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.501395    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8687da3a-7a11-4a9f-9f72-95dc214f413f" containerName="main"
Sep 05 20:03:19.501466 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.501401    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8687da3a-7a11-4a9f-9f72-95dc214f413f" containerName="main"
Sep 05 20:03:19.501466 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.501411    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56770103-b7ef-4c4c-8868-03e81ec44ad5" containerName="istio-proxy"
Sep 05 20:03:19.501466 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.501417    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="56770103-b7ef-4c4c-8868-03e81ec44ad5" containerName="istio-proxy"
Sep 05 20:03:19.501466 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.501423    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed33b879-a2ca-44a4-8377-fdc18928c09b" containerName="payload-processing"
Sep 05 20:03:19.501466 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.501428    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed33b879-a2ca-44a4-8377-fdc18928c09b" containerName="payload-processing"
Sep 05 20:03:19.501466 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.501437    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc4bff8-1fc7-4f3c-bb3a-99969447e318" containerName="main"
Sep 05 20:03:19.501466 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.501442    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc4bff8-1fc7-4f3c-bb3a-99969447e318" containerName="main"
Sep 05 20:03:19.501466 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.501448    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56770103-b7ef-4c4c-8868-03e81ec44ad5" containerName="istio-proxy"
Sep 05 20:03:19.501466 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.501453    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="56770103-b7ef-4c4c-8868-03e81ec44ad5" containerName="istio-proxy"
Sep 05 20:03:19.501466 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.501459    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91b1886c-1172-40f9-b429-c216c381853b" containerName="revoke-keys"
Sep 05 20:03:19.501466 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.501464    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="91b1886c-1172-40f9-b429-c216c381853b" containerName="revoke-keys"
Sep 05 20:03:19.501880 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.501523    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="56770103-b7ef-4c4c-8868-03e81ec44ad5" containerName="istio-proxy"
Sep 05 20:03:19.501880 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.501531    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="56770103-b7ef-4c4c-8868-03e81ec44ad5" containerName="istio-proxy"
Sep 05 20:03:19.501880 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.501537    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed33b879-a2ca-44a4-8377-fdc18928c09b" containerName="payload-processing"
Sep 05 20:03:19.501880 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.501543    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dc4bff8-1fc7-4f3c-bb3a-99969447e318" containerName="main"
Sep 05 20:03:19.501880 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.501551    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="00416952-66bf-46d7-84bb-617343832ad5" containerName="payload-pre-processing"
Sep 05 20:03:19.501880 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.501557    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8687da3a-7a11-4a9f-9f72-95dc214f413f" containerName="main"
Sep 05 20:03:19.501880 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.501562    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d96c4ef-3daa-42ae-af20-d4640421f102" containerName="payload-processing"
Sep 05 20:03:19.501880 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.501569    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="91b1886c-1172-40f9-b429-c216c381853b" containerName="revoke-keys"
Sep 05 20:03:19.505936 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.505919    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk"
Sep 05 20:03:19.508152 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.508099    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fedc2772\""
Sep 05 20:03:19.508270 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.508233    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fedc2772-dockercfg-4sfqs\""
Sep 05 20:03:19.516751 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.516728    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk"]
Sep 05 20:03:19.537448 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.537414    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs"]
Sep 05 20:03:19.540927 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.540910    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs"
Sep 05 20:03:19.552303 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.552278    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs"]
Sep 05 20:03:19.563756 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.563728    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tm6t\" (UniqueName: \"kubernetes.io/projected/44a510ef-1998-46aa-814f-bbe19bea757c-kube-api-access-7tm6t\") pod \"payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk\" (UID: \"44a510ef-1998-46aa-814f-bbe19bea757c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk"
Sep 05 20:03:19.563871 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.563777    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhdzc\" (UniqueName: \"kubernetes.io/projected/322b686c-dddc-4880-992f-afa906a71782-kube-api-access-fhdzc\") pod \"payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs\" (UID: \"322b686c-dddc-4880-992f-afa906a71782\") " pod="openshift-ingress/payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs"
Sep 05 20:03:19.563871 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.563841    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322b686c-dddc-4880-992f-afa906a71782-plugins-config-volume\") pod \"payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs\" (UID: \"322b686c-dddc-4880-992f-afa906a71782\") " pod="openshift-ingress/payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs"
Sep 05 20:03:19.563945 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.563886    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a510ef-1998-46aa-814f-bbe19bea757c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk\" (UID: \"44a510ef-1998-46aa-814f-bbe19bea757c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk"
Sep 05 20:03:19.664580 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.664530    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tm6t\" (UniqueName: \"kubernetes.io/projected/44a510ef-1998-46aa-814f-bbe19bea757c-kube-api-access-7tm6t\") pod \"payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk\" (UID: \"44a510ef-1998-46aa-814f-bbe19bea757c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk"
Sep 05 20:03:19.664780 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.664602    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhdzc\" (UniqueName: \"kubernetes.io/projected/322b686c-dddc-4880-992f-afa906a71782-kube-api-access-fhdzc\") pod \"payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs\" (UID: \"322b686c-dddc-4880-992f-afa906a71782\") " pod="openshift-ingress/payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs"
Sep 05 20:03:19.664780 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.664633    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322b686c-dddc-4880-992f-afa906a71782-plugins-config-volume\") pod \"payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs\" (UID: \"322b686c-dddc-4880-992f-afa906a71782\") " pod="openshift-ingress/payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs"
Sep 05 20:03:19.664780 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.664665    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a510ef-1998-46aa-814f-bbe19bea757c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk\" (UID: \"44a510ef-1998-46aa-814f-bbe19bea757c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk"
Sep 05 20:03:19.665546 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.665515    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a510ef-1998-46aa-814f-bbe19bea757c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk\" (UID: \"44a510ef-1998-46aa-814f-bbe19bea757c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk"
Sep 05 20:03:19.665953 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.665932    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322b686c-dddc-4880-992f-afa906a71782-plugins-config-volume\") pod \"payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs\" (UID: \"322b686c-dddc-4880-992f-afa906a71782\") " pod="openshift-ingress/payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs"
Sep 05 20:03:19.673308 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.673283    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tm6t\" (UniqueName: \"kubernetes.io/projected/44a510ef-1998-46aa-814f-bbe19bea757c-kube-api-access-7tm6t\") pod \"payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk\" (UID: \"44a510ef-1998-46aa-814f-bbe19bea757c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk"
Sep 05 20:03:19.673431 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.673390    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhdzc\" (UniqueName: \"kubernetes.io/projected/322b686c-dddc-4880-992f-afa906a71782-kube-api-access-fhdzc\") pod \"payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs\" (UID: \"322b686c-dddc-4880-992f-afa906a71782\") " pod="openshift-ingress/payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs"
Sep 05 20:03:19.817298 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.817208    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk"
Sep 05 20:03:19.852594 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:19.852555    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs"
Sep 05 20:03:20.095790 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:20.095764    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk"]
Sep 05 20:03:20.096851 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:03:20.096826    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a510ef_1998_46aa_814f_bbe19bea757c.slice/crio-1f27862ed092f31a508a37f91e76197aa4295b4d890884027c8968a8adc5c0ce WatchSource:0}: Error finding container 1f27862ed092f31a508a37f91e76197aa4295b4d890884027c8968a8adc5c0ce: Status 404 returned error can't find the container with id 1f27862ed092f31a508a37f91e76197aa4295b4d890884027c8968a8adc5c0ce
Sep 05 20:03:20.140511 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:20.140481    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs"]
Sep 05 20:03:20.140930 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:03:20.140908    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322b686c_dddc_4880_992f_afa906a71782.slice/crio-2f308d81d562c67f08da53bfc8bd2ad40159dd34dc8ca5add0b4c7bf089da215 WatchSource:0}: Error finding container 2f308d81d562c67f08da53bfc8bd2ad40159dd34dc8ca5add0b4c7bf089da215: Status 404 returned error can't find the container with id 2f308d81d562c67f08da53bfc8bd2ad40159dd34dc8ca5add0b4c7bf089da215
Sep 05 20:03:20.928443 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:20.928410    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs" event={"ID":"322b686c-dddc-4880-992f-afa906a71782","Type":"ContainerStarted","Data":"89db8382ecdbd9e5dcca2280f1bde48cbe617317e7be88fd745e46b964268192"}
Sep 05 20:03:20.928898 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:20.928461    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs" event={"ID":"322b686c-dddc-4880-992f-afa906a71782","Type":"ContainerStarted","Data":"2f308d81d562c67f08da53bfc8bd2ad40159dd34dc8ca5add0b4c7bf089da215"}
Sep 05 20:03:20.928898 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:20.928491    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs"
Sep 05 20:03:20.930144 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:20.930098    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk" event={"ID":"44a510ef-1998-46aa-814f-bbe19bea757c","Type":"ContainerStarted","Data":"c847f2f19179d189e3abc6e142a053d023b05eb693ce201848febd9691d6e9eb"}
Sep 05 20:03:20.930281 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:20.930170    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk" event={"ID":"44a510ef-1998-46aa-814f-bbe19bea757c","Type":"ContainerStarted","Data":"1f27862ed092f31a508a37f91e76197aa4295b4d890884027c8968a8adc5c0ce"}
Sep 05 20:03:20.930281 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:20.930199    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk"
Sep 05 20:03:20.949522 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:20.949473    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs" podStartSLOduration=1.949457308 podStartE2EDuration="1.949457308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:03:20.947885742 +0000 UTC m=+1199.977517648" watchObservedRunningTime="2026-09-05 20:03:20.949457308 +0000 UTC m=+1199.979089283"
Sep 05 20:03:20.967322 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:20.967261    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk" podStartSLOduration=1.967241385 podStartE2EDuration="1.967241385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:03:20.963846399 +0000 UTC m=+1199.993478304" watchObservedRunningTime="2026-09-05 20:03:20.967241385 +0000 UTC m=+1199.996873293"
Sep 05 20:03:21.451694 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.451552    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw"]
Sep 05 20:03:21.452202 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.451950    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw" podUID="87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa" containerName="payload-processing" containerID="cri-o://452c84c07df8fe8ea71d50a609330e113dab908becea739d1b98b5cbf231fc3f" gracePeriod=30
Sep 05 20:03:21.459685 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.459659    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9"]
Sep 05 20:03:21.460405 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.460347    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9" podUID="22b964ab-a361-4c66-af33-03426c04b2e6" containerName="payload-pre-processing" containerID="cri-o://fbd3ca49f64be86b230e8314629c208bf06d578981db8a3976d930588a1c1f1a" gracePeriod=30
Sep 05 20:03:21.517137 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.517029    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf9tj_f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b/ovn-acl-logging/0.log"
Sep 05 20:03:21.520699 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.520669    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf9tj_f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b/ovn-acl-logging/0.log"
Sep 05 20:03:21.762533 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.762504    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw"
Sep 05 20:03:21.766465 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.766438    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9"
Sep 05 20:03:21.785023 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.784983    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b964ab-a361-4c66-af33-03426c04b2e6-plugins-config-volume\") pod \"22b964ab-a361-4c66-af33-03426c04b2e6\" (UID: \"22b964ab-a361-4c66-af33-03426c04b2e6\") "
Sep 05 20:03:21.785227 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.785061    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkrr6\" (UniqueName: \"kubernetes.io/projected/22b964ab-a361-4c66-af33-03426c04b2e6-kube-api-access-kkrr6\") pod \"22b964ab-a361-4c66-af33-03426c04b2e6\" (UID: \"22b964ab-a361-4c66-af33-03426c04b2e6\") "
Sep 05 20:03:21.785227 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.785092    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x22hq\" (UniqueName: \"kubernetes.io/projected/87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa-kube-api-access-x22hq\") pod \"87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa\" (UID: \"87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa\") "
Sep 05 20:03:21.785227 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.785148    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa-plugins-config-volume\") pod \"87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa\" (UID: \"87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa\") "
Sep 05 20:03:21.785474 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.785444    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b964ab-a361-4c66-af33-03426c04b2e6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22b964ab-a361-4c66-af33-03426c04b2e6" (UID: "22b964ab-a361-4c66-af33-03426c04b2e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:03:21.785838 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.785813    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa" (UID: "87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:03:21.787602 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.787578    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b964ab-a361-4c66-af33-03426c04b2e6-kube-api-access-kkrr6" (OuterVolumeSpecName: "kube-api-access-kkrr6") pod "22b964ab-a361-4c66-af33-03426c04b2e6" (UID: "22b964ab-a361-4c66-af33-03426c04b2e6"). InnerVolumeSpecName "kube-api-access-kkrr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:03:21.787695 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.787648    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa-kube-api-access-x22hq" (OuterVolumeSpecName: "kube-api-access-x22hq") pod "87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa" (UID: "87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa"). InnerVolumeSpecName "kube-api-access-x22hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:03:21.886366 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.886331    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkrr6\" (UniqueName: \"kubernetes.io/projected/22b964ab-a361-4c66-af33-03426c04b2e6-kube-api-access-kkrr6\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:03:21.886366 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.886362    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x22hq\" (UniqueName: \"kubernetes.io/projected/87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa-kube-api-access-x22hq\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:03:21.886366 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.886371    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:03:21.886595 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.886380    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b964ab-a361-4c66-af33-03426c04b2e6-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:03:21.937970 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.937940    2583 generic.go:356] "Generic (PLEG): container finished" podID="87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa" containerID="452c84c07df8fe8ea71d50a609330e113dab908becea739d1b98b5cbf231fc3f" exitCode=0
Sep 05 20:03:21.938395 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.937985    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw"
Sep 05 20:03:21.938395 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.938030    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw" event={"ID":"87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa","Type":"ContainerDied","Data":"452c84c07df8fe8ea71d50a609330e113dab908becea739d1b98b5cbf231fc3f"}
Sep 05 20:03:21.938395 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.938084    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw" event={"ID":"87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa","Type":"ContainerDied","Data":"f524cbd7580c2e03c1b868192e5be7810b48e476245d357078177371be4e833f"}
Sep 05 20:03:21.938395 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.938147    2583 scope.go:117] "RemoveContainer" containerID="452c84c07df8fe8ea71d50a609330e113dab908becea739d1b98b5cbf231fc3f"
Sep 05 20:03:21.939560 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.939539    2583 generic.go:356] "Generic (PLEG): container finished" podID="22b964ab-a361-4c66-af33-03426c04b2e6" containerID="fbd3ca49f64be86b230e8314629c208bf06d578981db8a3976d930588a1c1f1a" exitCode=0
Sep 05 20:03:21.939667 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.939569    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9" event={"ID":"22b964ab-a361-4c66-af33-03426c04b2e6","Type":"ContainerDied","Data":"fbd3ca49f64be86b230e8314629c208bf06d578981db8a3976d930588a1c1f1a"}
Sep 05 20:03:21.939667 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.939599    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9"
Sep 05 20:03:21.939667 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.939618    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9" event={"ID":"22b964ab-a361-4c66-af33-03426c04b2e6","Type":"ContainerDied","Data":"967380b26ccac0e8e32781fe441080917ed1b97f52786f95d5abacbf03432106"}
Sep 05 20:03:21.947927 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.947902    2583 scope.go:117] "RemoveContainer" containerID="452c84c07df8fe8ea71d50a609330e113dab908becea739d1b98b5cbf231fc3f"
Sep 05 20:03:21.948394 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:03:21.948263    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"452c84c07df8fe8ea71d50a609330e113dab908becea739d1b98b5cbf231fc3f\": container with ID starting with 452c84c07df8fe8ea71d50a609330e113dab908becea739d1b98b5cbf231fc3f not found: ID does not exist" containerID="452c84c07df8fe8ea71d50a609330e113dab908becea739d1b98b5cbf231fc3f"
Sep 05 20:03:21.948394 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.948287    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452c84c07df8fe8ea71d50a609330e113dab908becea739d1b98b5cbf231fc3f"} err="failed to get container status \"452c84c07df8fe8ea71d50a609330e113dab908becea739d1b98b5cbf231fc3f\": rpc error: code = NotFound desc = could not find container \"452c84c07df8fe8ea71d50a609330e113dab908becea739d1b98b5cbf231fc3f\": container with ID starting with 452c84c07df8fe8ea71d50a609330e113dab908becea739d1b98b5cbf231fc3f not found: ID does not exist"
Sep 05 20:03:21.948394 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.948303    2583 scope.go:117] "RemoveContainer" containerID="fbd3ca49f64be86b230e8314629c208bf06d578981db8a3976d930588a1c1f1a"
Sep 05 20:03:21.956923 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.956907    2583 scope.go:117] "RemoveContainer" containerID="fbd3ca49f64be86b230e8314629c208bf06d578981db8a3976d930588a1c1f1a"
Sep 05 20:03:21.957302 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:03:21.957279    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbd3ca49f64be86b230e8314629c208bf06d578981db8a3976d930588a1c1f1a\": container with ID starting with fbd3ca49f64be86b230e8314629c208bf06d578981db8a3976d930588a1c1f1a not found: ID does not exist" containerID="fbd3ca49f64be86b230e8314629c208bf06d578981db8a3976d930588a1c1f1a"
Sep 05 20:03:21.957368 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.957311    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbd3ca49f64be86b230e8314629c208bf06d578981db8a3976d930588a1c1f1a"} err="failed to get container status \"fbd3ca49f64be86b230e8314629c208bf06d578981db8a3976d930588a1c1f1a\": rpc error: code = NotFound desc = could not find container \"fbd3ca49f64be86b230e8314629c208bf06d578981db8a3976d930588a1c1f1a\": container with ID starting with fbd3ca49f64be86b230e8314629c208bf06d578981db8a3976d930588a1c1f1a not found: ID does not exist"
Sep 05 20:03:21.964272 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.964245    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9"]
Sep 05 20:03:21.969950 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.969924    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1dc7b1-77698b8fb6-kq5s9"]
Sep 05 20:03:21.977878 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.977850    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw"]
Sep 05 20:03:21.982405 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:21.982338    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1dc7b1-7b995595-cwqkw"]
Sep 05 20:03:23.435371 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:23.435337    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22b964ab-a361-4c66-af33-03426c04b2e6" path="/var/lib/kubelet/pods/22b964ab-a361-4c66-af33-03426c04b2e6/volumes"
Sep 05 20:03:23.435739 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:23.435697    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa" path="/var/lib/kubelet/pods/87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa/volumes"
Sep 05 20:03:29.081220 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.081184    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"]
Sep 05 20:03:29.081791 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.081772    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22b964ab-a361-4c66-af33-03426c04b2e6" containerName="payload-pre-processing"
Sep 05 20:03:29.081831 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.081795    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="22b964ab-a361-4c66-af33-03426c04b2e6" containerName="payload-pre-processing"
Sep 05 20:03:29.081831 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.081821    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa" containerName="payload-processing"
Sep 05 20:03:29.081904 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.081831    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa" containerName="payload-processing"
Sep 05 20:03:29.081934 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.081921    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="22b964ab-a361-4c66-af33-03426c04b2e6" containerName="payload-pre-processing"
Sep 05 20:03:29.081968 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.081934    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="87a8f5b6-5861-49cb-81d2-8b71cdbc1eaa" containerName="payload-processing"
Sep 05 20:03:29.087028 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.087005    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.089274 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.089254    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-4d1885-openshift-default-dockercfg-d8ndk\""
Sep 05 20:03:29.095586 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.095214    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"]
Sep 05 20:03:29.147598 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.147561    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12890272-a09c-4655-a206-464c55195412-istio-token\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.147783 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.147614    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l98l\" (UniqueName: \"kubernetes.io/projected/12890272-a09c-4655-a206-464c55195412-kube-api-access-9l98l\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.147783 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.147643    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-credential-socket\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.147783 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.147686    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12890272-a09c-4655-a206-464c55195412-istio-podinfo\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.147783 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.147710    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-istio-data\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.147783 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.147779    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-workload-socket\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.147997 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.147803    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-workload-certs\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.147997 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.147858    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-istio-envoy\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.147997 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.147900    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12890272-a09c-4655-a206-464c55195412-istiod-ca-cert\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.248899 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.248856    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-workload-socket\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.248899 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.248900    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-workload-certs\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.249184 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.248930    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-istio-envoy\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.249184 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.248962    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12890272-a09c-4655-a206-464c55195412-istiod-ca-cert\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.249184 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.249004    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12890272-a09c-4655-a206-464c55195412-istio-token\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.249184 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.249045    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l98l\" (UniqueName: \"kubernetes.io/projected/12890272-a09c-4655-a206-464c55195412-kube-api-access-9l98l\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.249184 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.249071    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-credential-socket\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.249495 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.249152    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12890272-a09c-4655-a206-464c55195412-istio-podinfo\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.249495 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.249224    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-istio-data\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.249495 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.249359    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-workload-certs\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.249495 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.249405    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-workload-socket\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.249692 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.249612    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-istio-data\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.249692 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.249634    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-credential-socket\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.249918 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.249892    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12890272-a09c-4655-a206-464c55195412-istiod-ca-cert\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.251239 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.251214    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-istio-envoy\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.251716 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.251693    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12890272-a09c-4655-a206-464c55195412-istio-podinfo\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.257672 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.257645    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12890272-a09c-4655-a206-464c55195412-istio-token\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.257787 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.257687    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l98l\" (UniqueName: \"kubernetes.io/projected/12890272-a09c-4655-a206-464c55195412-kube-api-access-9l98l\") pod \"e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn\" (UID: \"12890272-a09c-4655-a206-464c55195412\") " pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.400684 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.400590    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:29.701088 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.701049    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"]
Sep 05 20:03:29.704214 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.704173    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 20:03:29.704336 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.704259    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 20:03:29.704336 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.704302    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 20:03:29.995198 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.995091    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn" event={"ID":"12890272-a09c-4655-a206-464c55195412","Type":"ContainerStarted","Data":"433e41c95469c3be61ac036ee1c2e97249f0376dec32ba4f60fe88792869a957"}
Sep 05 20:03:29.995198 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:29.995167    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn" event={"ID":"12890272-a09c-4655-a206-464c55195412","Type":"ContainerStarted","Data":"d48e834bf01f60c027e3e95eddddc2aed084ca0e22ae5d7a310888f938873ab8"}
Sep 05 20:03:30.016668 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:30.016615    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn" podStartSLOduration=1.016599598 podStartE2EDuration="1.016599598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:03:30.013509583 +0000 UTC m=+1209.043141488" watchObservedRunningTime="2026-09-05 20:03:30.016599598 +0000 UTC m=+1209.046231497"
Sep 05 20:03:30.401287 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:30.401247    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:30.406149 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:30.406120    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:31.001332 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:31.001294    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:31.002439 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:31.002420    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:31.941686 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:31.941653    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk"
Sep 05 20:03:31.942052 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:31.941812    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs"
Sep 05 20:03:33.792552 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:33.792515    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl"]
Sep 05 20:03:33.796276 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:33.796260    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl"
Sep 05 20:03:33.798534 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:33.798514    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-4d1885\""
Sep 05 20:03:33.798630 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:33.798615    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-4d1885-dockercfg-7zs9f\""
Sep 05 20:03:33.811862 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:33.811838    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl"]
Sep 05 20:03:33.839687 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:33.839659    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf"]
Sep 05 20:03:33.843209 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:33.843186    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf"
Sep 05 20:03:33.853255 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:33.853226    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf"]
Sep 05 20:03:33.891418 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:33.891382    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4cfd886-4a10-4b97-b3bf-e7bd86e8b740-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf\" (UID: \"e4cfd886-4a10-4b97-b3bf-e7bd86e8b740\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf"
Sep 05 20:03:33.891576 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:33.891448    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be0ebded-91d9-40b0-87cd-52a3a9093edd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl\" (UID: \"be0ebded-91d9-40b0-87cd-52a3a9093edd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl"
Sep 05 20:03:33.891576 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:33.891470    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcx8k\" (UniqueName: \"kubernetes.io/projected/be0ebded-91d9-40b0-87cd-52a3a9093edd-kube-api-access-wcx8k\") pod \"payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl\" (UID: \"be0ebded-91d9-40b0-87cd-52a3a9093edd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl"
Sep 05 20:03:33.891670 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:33.891589    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp9fg\" (UniqueName: \"kubernetes.io/projected/e4cfd886-4a10-4b97-b3bf-e7bd86e8b740-kube-api-access-qp9fg\") pod \"payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf\" (UID: \"e4cfd886-4a10-4b97-b3bf-e7bd86e8b740\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf"
Sep 05 20:03:33.992476 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:33.992439    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp9fg\" (UniqueName: \"kubernetes.io/projected/e4cfd886-4a10-4b97-b3bf-e7bd86e8b740-kube-api-access-qp9fg\") pod \"payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf\" (UID: \"e4cfd886-4a10-4b97-b3bf-e7bd86e8b740\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf"
Sep 05 20:03:33.992670 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:33.992516    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4cfd886-4a10-4b97-b3bf-e7bd86e8b740-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf\" (UID: \"e4cfd886-4a10-4b97-b3bf-e7bd86e8b740\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf"
Sep 05 20:03:33.992670 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:33.992568    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be0ebded-91d9-40b0-87cd-52a3a9093edd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl\" (UID: \"be0ebded-91d9-40b0-87cd-52a3a9093edd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl"
Sep 05 20:03:33.992670 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:33.992664    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcx8k\" (UniqueName: \"kubernetes.io/projected/be0ebded-91d9-40b0-87cd-52a3a9093edd-kube-api-access-wcx8k\") pod \"payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl\" (UID: \"be0ebded-91d9-40b0-87cd-52a3a9093edd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl"
Sep 05 20:03:33.993313 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:33.993287    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4cfd886-4a10-4b97-b3bf-e7bd86e8b740-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf\" (UID: \"e4cfd886-4a10-4b97-b3bf-e7bd86e8b740\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf"
Sep 05 20:03:33.993423 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:33.993342    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be0ebded-91d9-40b0-87cd-52a3a9093edd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl\" (UID: \"be0ebded-91d9-40b0-87cd-52a3a9093edd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl"
Sep 05 20:03:34.001147 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:34.001091    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp9fg\" (UniqueName: \"kubernetes.io/projected/e4cfd886-4a10-4b97-b3bf-e7bd86e8b740-kube-api-access-qp9fg\") pod \"payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf\" (UID: \"e4cfd886-4a10-4b97-b3bf-e7bd86e8b740\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf"
Sep 05 20:03:34.001241 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:34.001178    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcx8k\" (UniqueName: \"kubernetes.io/projected/be0ebded-91d9-40b0-87cd-52a3a9093edd-kube-api-access-wcx8k\") pod \"payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl\" (UID: \"be0ebded-91d9-40b0-87cd-52a3a9093edd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl"
Sep 05 20:03:34.107444 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:34.107412    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl"
Sep 05 20:03:34.154009 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:34.153971    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf"
Sep 05 20:03:34.377757 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:34.377730    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl"]
Sep 05 20:03:34.379219 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:03:34.379189    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0ebded_91d9_40b0_87cd_52a3a9093edd.slice/crio-734a64b59bb2dd64434b49af4da5a670dddb116ed77de121cf0eac0877708079 WatchSource:0}: Error finding container 734a64b59bb2dd64434b49af4da5a670dddb116ed77de121cf0eac0877708079: Status 404 returned error can't find the container with id 734a64b59bb2dd64434b49af4da5a670dddb116ed77de121cf0eac0877708079
Sep 05 20:03:34.441754 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:34.441731    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf"]
Sep 05 20:03:34.442541 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:03:34.442519    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4cfd886_4a10_4b97_b3bf_e7bd86e8b740.slice/crio-44fa6c0814b82911c40ab50d7e76f85989f5eb2e7b8fb7342793b6172d4e4911 WatchSource:0}: Error finding container 44fa6c0814b82911c40ab50d7e76f85989f5eb2e7b8fb7342793b6172d4e4911: Status 404 returned error can't find the container with id 44fa6c0814b82911c40ab50d7e76f85989f5eb2e7b8fb7342793b6172d4e4911
Sep 05 20:03:35.029844 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:35.029810    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl" event={"ID":"be0ebded-91d9-40b0-87cd-52a3a9093edd","Type":"ContainerStarted","Data":"fc0e7064fc100cc6da4913c21516481d986f983c51fcb7dedb7dc4043d66fcf0"}
Sep 05 20:03:35.030314 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:35.029869    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl" event={"ID":"be0ebded-91d9-40b0-87cd-52a3a9093edd","Type":"ContainerStarted","Data":"734a64b59bb2dd64434b49af4da5a670dddb116ed77de121cf0eac0877708079"}
Sep 05 20:03:35.030314 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:35.029973    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl"
Sep 05 20:03:35.031690 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:35.031666    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf" event={"ID":"e4cfd886-4a10-4b97-b3bf-e7bd86e8b740","Type":"ContainerStarted","Data":"1fdbb299ee8d2887b29166223d5839815c94ecfd8a7e9deb29fa2a0653fdab90"}
Sep 05 20:03:35.031824 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:35.031708    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf" event={"ID":"e4cfd886-4a10-4b97-b3bf-e7bd86e8b740","Type":"ContainerStarted","Data":"44fa6c0814b82911c40ab50d7e76f85989f5eb2e7b8fb7342793b6172d4e4911"}
Sep 05 20:03:35.031824 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:35.031735    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf"
Sep 05 20:03:35.048059 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:35.048011    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl" podStartSLOduration=2.047995105 podStartE2EDuration="2.047995105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:03:35.045212521 +0000 UTC m=+1214.074844428" watchObservedRunningTime="2026-09-05 20:03:35.047995105 +0000 UTC m=+1214.077627011"
Sep 05 20:03:35.059985 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:35.059943    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf" podStartSLOduration=2.059929196 podStartE2EDuration="2.059929196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:03:35.059459397 +0000 UTC m=+1214.089091304" watchObservedRunningTime="2026-09-05 20:03:35.059929196 +0000 UTC m=+1214.089561100"
Sep 05 20:03:46.044932 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.044896    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf"
Sep 05 20:03:46.045333 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.045069    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl"
Sep 05 20:03:46.226572 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.226529    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"]
Sep 05 20:03:46.231070 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.231047    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:03:46.233708 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.233682    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-0741a0\"/\"kube-root-ca.crt\""
Sep 05 20:03:46.233836 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.233687    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-0741a0\"/\"default-dockercfg-k5b98\""
Sep 05 20:03:46.233836 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.233687    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-0741a0\"/\"openshift-service-ca.crt\""
Sep 05 20:03:46.233836 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.233691    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-0741a0\"/\"noten-model-0741a0-kserve-self-signed-certs\""
Sep 05 20:03:46.242809 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.242787    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"]
Sep 05 20:03:46.324650 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.324613    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d725dd23-0c45-4bcf-951c-39883f21a95d-model-cache\") pod \"noten-model-0741a0-kserve-7c54c56cbc-v6fhq\" (UID: \"d725dd23-0c45-4bcf-951c-39883f21a95d\") " pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:03:46.324903 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.324705    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d725dd23-0c45-4bcf-951c-39883f21a95d-tmp-dir\") pod \"noten-model-0741a0-kserve-7c54c56cbc-v6fhq\" (UID: \"d725dd23-0c45-4bcf-951c-39883f21a95d\") " pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:03:46.324903 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.324727    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d725dd23-0c45-4bcf-951c-39883f21a95d-dshm\") pod \"noten-model-0741a0-kserve-7c54c56cbc-v6fhq\" (UID: \"d725dd23-0c45-4bcf-951c-39883f21a95d\") " pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:03:46.324903 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.324745    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d725dd23-0c45-4bcf-951c-39883f21a95d-tls-certs\") pod \"noten-model-0741a0-kserve-7c54c56cbc-v6fhq\" (UID: \"d725dd23-0c45-4bcf-951c-39883f21a95d\") " pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:03:46.324903 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.324769    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brnnf\" (UniqueName: \"kubernetes.io/projected/d725dd23-0c45-4bcf-951c-39883f21a95d-kube-api-access-brnnf\") pod \"noten-model-0741a0-kserve-7c54c56cbc-v6fhq\" (UID: \"d725dd23-0c45-4bcf-951c-39883f21a95d\") " pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:03:46.324903 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.324805    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d725dd23-0c45-4bcf-951c-39883f21a95d-home\") pod \"noten-model-0741a0-kserve-7c54c56cbc-v6fhq\" (UID: \"d725dd23-0c45-4bcf-951c-39883f21a95d\") " pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:03:46.425878 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.425840    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d725dd23-0c45-4bcf-951c-39883f21a95d-model-cache\") pod \"noten-model-0741a0-kserve-7c54c56cbc-v6fhq\" (UID: \"d725dd23-0c45-4bcf-951c-39883f21a95d\") " pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:03:46.426073 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.425913    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d725dd23-0c45-4bcf-951c-39883f21a95d-tmp-dir\") pod \"noten-model-0741a0-kserve-7c54c56cbc-v6fhq\" (UID: \"d725dd23-0c45-4bcf-951c-39883f21a95d\") " pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:03:46.426073 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.425933    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d725dd23-0c45-4bcf-951c-39883f21a95d-dshm\") pod \"noten-model-0741a0-kserve-7c54c56cbc-v6fhq\" (UID: \"d725dd23-0c45-4bcf-951c-39883f21a95d\") " pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:03:46.426073 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.425949    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d725dd23-0c45-4bcf-951c-39883f21a95d-tls-certs\") pod \"noten-model-0741a0-kserve-7c54c56cbc-v6fhq\" (UID: \"d725dd23-0c45-4bcf-951c-39883f21a95d\") " pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:03:46.426073 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.425967    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brnnf\" (UniqueName: \"kubernetes.io/projected/d725dd23-0c45-4bcf-951c-39883f21a95d-kube-api-access-brnnf\") pod \"noten-model-0741a0-kserve-7c54c56cbc-v6fhq\" (UID: \"d725dd23-0c45-4bcf-951c-39883f21a95d\") " pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:03:46.426073 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.425990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d725dd23-0c45-4bcf-951c-39883f21a95d-home\") pod \"noten-model-0741a0-kserve-7c54c56cbc-v6fhq\" (UID: \"d725dd23-0c45-4bcf-951c-39883f21a95d\") " pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:03:46.426395 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.426364    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d725dd23-0c45-4bcf-951c-39883f21a95d-model-cache\") pod \"noten-model-0741a0-kserve-7c54c56cbc-v6fhq\" (UID: \"d725dd23-0c45-4bcf-951c-39883f21a95d\") " pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:03:46.426445 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.426390    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d725dd23-0c45-4bcf-951c-39883f21a95d-tmp-dir\") pod \"noten-model-0741a0-kserve-7c54c56cbc-v6fhq\" (UID: \"d725dd23-0c45-4bcf-951c-39883f21a95d\") " pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:03:46.426496 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.426470    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d725dd23-0c45-4bcf-951c-39883f21a95d-home\") pod \"noten-model-0741a0-kserve-7c54c56cbc-v6fhq\" (UID: \"d725dd23-0c45-4bcf-951c-39883f21a95d\") " pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:03:46.428249 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.428227    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d725dd23-0c45-4bcf-951c-39883f21a95d-dshm\") pod \"noten-model-0741a0-kserve-7c54c56cbc-v6fhq\" (UID: \"d725dd23-0c45-4bcf-951c-39883f21a95d\") " pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:03:46.428554 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.428530    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d725dd23-0c45-4bcf-951c-39883f21a95d-tls-certs\") pod \"noten-model-0741a0-kserve-7c54c56cbc-v6fhq\" (UID: \"d725dd23-0c45-4bcf-951c-39883f21a95d\") " pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:03:46.434438 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.434412    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brnnf\" (UniqueName: \"kubernetes.io/projected/d725dd23-0c45-4bcf-951c-39883f21a95d-kube-api-access-brnnf\") pod \"noten-model-0741a0-kserve-7c54c56cbc-v6fhq\" (UID: \"d725dd23-0c45-4bcf-951c-39883f21a95d\") " pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:03:46.542735 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.542699    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:03:46.699911 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.699860    2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": read tcp 10.134.0.2:60416->10.134.0.79:15021: read: connection reset by peer" start-of-body=
Sep 05 20:03:46.700062 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.699933    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn" podUID="12890272-a09c-4655-a206-464c55195412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": read tcp 10.134.0.2:60416->10.134.0.79:15021: read: connection reset by peer"
Sep 05 20:03:46.837522 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:46.835529    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"]
Sep 05 20:03:46.841340 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:03:46.841309    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd725dd23_0c45_4bcf_951c_39883f21a95d.slice/crio-d23175e62bbb77fabec341e558e75fbaf32617224457e509843618a964e9642d WatchSource:0}: Error finding container d23175e62bbb77fabec341e558e75fbaf32617224457e509843618a964e9642d: Status 404 returned error can't find the container with id d23175e62bbb77fabec341e558e75fbaf32617224457e509843618a964e9642d
Sep 05 20:03:47.124989 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:47.124892    2583 generic.go:356] "Generic (PLEG): container finished" podID="12890272-a09c-4655-a206-464c55195412" containerID="433e41c95469c3be61ac036ee1c2e97249f0376dec32ba4f60fe88792869a957" exitCode=0
Sep 05 20:03:47.124989 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:47.124968    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn" event={"ID":"12890272-a09c-4655-a206-464c55195412","Type":"ContainerDied","Data":"433e41c95469c3be61ac036ee1c2e97249f0376dec32ba4f60fe88792869a957"}
Sep 05 20:03:47.125529 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:47.125512    2583 scope.go:117] "RemoveContainer" containerID="433e41c95469c3be61ac036ee1c2e97249f0376dec32ba4f60fe88792869a957"
Sep 05 20:03:47.126697 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:47.126669    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 20:03:47.126853 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:47.126741    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 20:03:47.126853 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:47.126777    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 20:03:47.126853 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:47.126800    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq" event={"ID":"d725dd23-0c45-4bcf-951c-39883f21a95d","Type":"ContainerStarted","Data":"e7ae6a59aa38581e3fe7e282de4b8a5861b4c44161d6f1f680ce4c5c14a0ab40"}
Sep 05 20:03:47.126853 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:47.126840    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq" event={"ID":"d725dd23-0c45-4bcf-951c-39883f21a95d","Type":"ContainerStarted","Data":"d23175e62bbb77fabec341e558e75fbaf32617224457e509843618a964e9642d"}
Sep 05 20:03:47.174253 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:47.173678    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq" podStartSLOduration=1.173657635 podStartE2EDuration="1.173657635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:03:47.172200675 +0000 UTC m=+1226.201832581" watchObservedRunningTime="2026-09-05 20:03:47.173657635 +0000 UTC m=+1226.203289542"
Sep 05 20:03:48.134560 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:48.134520    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn" event={"ID":"12890272-a09c-4655-a206-464c55195412","Type":"ContainerStarted","Data":"b1a854d1809007fa2ceff39deded1d1d41dd09e003c75c7ba32adc975d7ab811"}
Sep 05 20:03:48.401584 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:48.401490    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:48.401788 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:48.401762    2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 05 20:03:48.401860 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:48.401825    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn" podUID="12890272-a09c-4655-a206-464c55195412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 05 20:03:49.401029 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:49.400993    2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 05 20:03:49.401444 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:49.401047    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn" podUID="12890272-a09c-4655-a206-464c55195412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 05 20:03:50.401515 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:50.401478    2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 05 20:03:50.401903 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:50.401533    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn" podUID="12890272-a09c-4655-a206-464c55195412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 05 20:03:51.401734 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:51.401695    2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 05 20:03:51.402162 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:51.401752    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn" podUID="12890272-a09c-4655-a206-464c55195412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 05 20:03:52.401757 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:52.401719    2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 05 20:03:52.402179 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:52.401791    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn" podUID="12890272-a09c-4655-a206-464c55195412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 05 20:03:53.401751 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:53.401711    2583 patch_prober.go:28] interesting pod/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 05 20:03:53.401924 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:53.401773    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn" podUID="12890272-a09c-4655-a206-464c55195412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 05 20:03:54.405714 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:54.405683    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:54.406173 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:54.406137    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:54.406907 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:54.406885    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:03:56.543827 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:56.543792    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:03:56.544244 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:56.543841    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:03:56.556502 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:56.556476    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:03:57.209493 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:03:57.209461    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:04:00.472728 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:00.472681    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs"]
Sep 05 20:04:00.473218 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:00.472932    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs" podUID="322b686c-dddc-4880-992f-afa906a71782" containerName="payload-processing" containerID="cri-o://89db8382ecdbd9e5dcca2280f1bde48cbe617317e7be88fd745e46b964268192" gracePeriod=30
Sep 05 20:04:00.476517 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:00.476494    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk"]
Sep 05 20:04:00.476821 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:00.476687    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk" podUID="44a510ef-1998-46aa-814f-bbe19bea757c" containerName="payload-pre-processing" containerID="cri-o://c847f2f19179d189e3abc6e142a053d023b05eb693ce201848febd9691d6e9eb" gracePeriod=30
Sep 05 20:04:00.734019 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:00.733956    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs"
Sep 05 20:04:00.775732 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:00.775707    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk"
Sep 05 20:04:00.872054 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:00.872018    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322b686c-dddc-4880-992f-afa906a71782-plugins-config-volume\") pod \"322b686c-dddc-4880-992f-afa906a71782\" (UID: \"322b686c-dddc-4880-992f-afa906a71782\") "
Sep 05 20:04:00.872249 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:00.872072    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhdzc\" (UniqueName: \"kubernetes.io/projected/322b686c-dddc-4880-992f-afa906a71782-kube-api-access-fhdzc\") pod \"322b686c-dddc-4880-992f-afa906a71782\" (UID: \"322b686c-dddc-4880-992f-afa906a71782\") "
Sep 05 20:04:00.872448 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:00.872413    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322b686c-dddc-4880-992f-afa906a71782-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "322b686c-dddc-4880-992f-afa906a71782" (UID: "322b686c-dddc-4880-992f-afa906a71782"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:04:00.874345 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:00.874325    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322b686c-dddc-4880-992f-afa906a71782-kube-api-access-fhdzc" (OuterVolumeSpecName: "kube-api-access-fhdzc") pod "322b686c-dddc-4880-992f-afa906a71782" (UID: "322b686c-dddc-4880-992f-afa906a71782"). InnerVolumeSpecName "kube-api-access-fhdzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:04:00.972795 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:00.972752    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tm6t\" (UniqueName: \"kubernetes.io/projected/44a510ef-1998-46aa-814f-bbe19bea757c-kube-api-access-7tm6t\") pod \"44a510ef-1998-46aa-814f-bbe19bea757c\" (UID: \"44a510ef-1998-46aa-814f-bbe19bea757c\") "
Sep 05 20:04:00.972989 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:00.972840    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a510ef-1998-46aa-814f-bbe19bea757c-plugins-config-volume\") pod \"44a510ef-1998-46aa-814f-bbe19bea757c\" (UID: \"44a510ef-1998-46aa-814f-bbe19bea757c\") "
Sep 05 20:04:00.973201 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:00.973180    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/322b686c-dddc-4880-992f-afa906a71782-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:04:00.973281 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:00.973203    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhdzc\" (UniqueName: \"kubernetes.io/projected/322b686c-dddc-4880-992f-afa906a71782-kube-api-access-fhdzc\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:04:00.973281 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:00.973193    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a510ef-1998-46aa-814f-bbe19bea757c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "44a510ef-1998-46aa-814f-bbe19bea757c" (UID: "44a510ef-1998-46aa-814f-bbe19bea757c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:04:00.974870 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:00.974846    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a510ef-1998-46aa-814f-bbe19bea757c-kube-api-access-7tm6t" (OuterVolumeSpecName: "kube-api-access-7tm6t") pod "44a510ef-1998-46aa-814f-bbe19bea757c" (UID: "44a510ef-1998-46aa-814f-bbe19bea757c"). InnerVolumeSpecName "kube-api-access-7tm6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:04:01.074370 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:01.074331    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tm6t\" (UniqueName: \"kubernetes.io/projected/44a510ef-1998-46aa-814f-bbe19bea757c-kube-api-access-7tm6t\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:04:01.074370 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:01.074369    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a510ef-1998-46aa-814f-bbe19bea757c-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:04:01.224589 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:01.224550    2583 generic.go:356] "Generic (PLEG): container finished" podID="44a510ef-1998-46aa-814f-bbe19bea757c" containerID="c847f2f19179d189e3abc6e142a053d023b05eb693ce201848febd9691d6e9eb" exitCode=0
Sep 05 20:04:01.224781 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:01.224597    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk"
Sep 05 20:04:01.224781 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:01.224642    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk" event={"ID":"44a510ef-1998-46aa-814f-bbe19bea757c","Type":"ContainerDied","Data":"c847f2f19179d189e3abc6e142a053d023b05eb693ce201848febd9691d6e9eb"}
Sep 05 20:04:01.224781 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:01.224705    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk" event={"ID":"44a510ef-1998-46aa-814f-bbe19bea757c","Type":"ContainerDied","Data":"1f27862ed092f31a508a37f91e76197aa4295b4d890884027c8968a8adc5c0ce"}
Sep 05 20:04:01.224781 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:01.224736    2583 scope.go:117] "RemoveContainer" containerID="c847f2f19179d189e3abc6e142a053d023b05eb693ce201848febd9691d6e9eb"
Sep 05 20:04:01.226114 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:01.226087    2583 generic.go:356] "Generic (PLEG): container finished" podID="322b686c-dddc-4880-992f-afa906a71782" containerID="89db8382ecdbd9e5dcca2280f1bde48cbe617317e7be88fd745e46b964268192" exitCode=0
Sep 05 20:04:01.226219 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:01.226155    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs"
Sep 05 20:04:01.226219 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:01.226173    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs" event={"ID":"322b686c-dddc-4880-992f-afa906a71782","Type":"ContainerDied","Data":"89db8382ecdbd9e5dcca2280f1bde48cbe617317e7be88fd745e46b964268192"}
Sep 05 20:04:01.226323 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:01.226223    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs" event={"ID":"322b686c-dddc-4880-992f-afa906a71782","Type":"ContainerDied","Data":"2f308d81d562c67f08da53bfc8bd2ad40159dd34dc8ca5add0b4c7bf089da215"}
Sep 05 20:04:01.234235 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:01.234218    2583 scope.go:117] "RemoveContainer" containerID="c847f2f19179d189e3abc6e142a053d023b05eb693ce201848febd9691d6e9eb"
Sep 05 20:04:01.234640 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:04:01.234616    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c847f2f19179d189e3abc6e142a053d023b05eb693ce201848febd9691d6e9eb\": container with ID starting with c847f2f19179d189e3abc6e142a053d023b05eb693ce201848febd9691d6e9eb not found: ID does not exist" containerID="c847f2f19179d189e3abc6e142a053d023b05eb693ce201848febd9691d6e9eb"
Sep 05 20:04:01.234732 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:01.234648    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c847f2f19179d189e3abc6e142a053d023b05eb693ce201848febd9691d6e9eb"} err="failed to get container status \"c847f2f19179d189e3abc6e142a053d023b05eb693ce201848febd9691d6e9eb\": rpc error: code = NotFound desc = could not find container \"c847f2f19179d189e3abc6e142a053d023b05eb693ce201848febd9691d6e9eb\": container with ID starting with c847f2f19179d189e3abc6e142a053d023b05eb693ce201848febd9691d6e9eb not found: ID does not exist"
Sep 05 20:04:01.234732 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:01.234663    2583 scope.go:117] "RemoveContainer" containerID="89db8382ecdbd9e5dcca2280f1bde48cbe617317e7be88fd745e46b964268192"
Sep 05 20:04:01.243438 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:01.243410    2583 scope.go:117] "RemoveContainer" containerID="89db8382ecdbd9e5dcca2280f1bde48cbe617317e7be88fd745e46b964268192"
Sep 05 20:04:01.243686 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:04:01.243664    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89db8382ecdbd9e5dcca2280f1bde48cbe617317e7be88fd745e46b964268192\": container with ID starting with 89db8382ecdbd9e5dcca2280f1bde48cbe617317e7be88fd745e46b964268192 not found: ID does not exist" containerID="89db8382ecdbd9e5dcca2280f1bde48cbe617317e7be88fd745e46b964268192"
Sep 05 20:04:01.243747 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:01.243698    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89db8382ecdbd9e5dcca2280f1bde48cbe617317e7be88fd745e46b964268192"} err="failed to get container status \"89db8382ecdbd9e5dcca2280f1bde48cbe617317e7be88fd745e46b964268192\": rpc error: code = NotFound desc = could not find container \"89db8382ecdbd9e5dcca2280f1bde48cbe617317e7be88fd745e46b964268192\": container with ID starting with 89db8382ecdbd9e5dcca2280f1bde48cbe617317e7be88fd745e46b964268192 not found: ID does not exist"
Sep 05 20:04:01.257598 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:01.257576    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs"]
Sep 05 20:04:01.275644 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:01.275612    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fedc2772-64d9885697-l8jvs"]
Sep 05 20:04:01.293744 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:01.293720    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk"]
Sep 05 20:04:01.302345 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:01.302322    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fedc2772-8cf787c5d-qjqdk"]
Sep 05 20:04:01.435149 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:01.435061    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="322b686c-dddc-4880-992f-afa906a71782" path="/var/lib/kubelet/pods/322b686c-dddc-4880-992f-afa906a71782/volumes"
Sep 05 20:04:01.435483 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:01.435469    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44a510ef-1998-46aa-814f-bbe19bea757c" path="/var/lib/kubelet/pods/44a510ef-1998-46aa-814f-bbe19bea757c/volumes"
Sep 05 20:04:02.432861 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.432820    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz"]
Sep 05 20:04:02.433449 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.433429    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44a510ef-1998-46aa-814f-bbe19bea757c" containerName="payload-pre-processing"
Sep 05 20:04:02.433532 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.433452    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="44a510ef-1998-46aa-814f-bbe19bea757c" containerName="payload-pre-processing"
Sep 05 20:04:02.433532 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.433471    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322b686c-dddc-4880-992f-afa906a71782" containerName="payload-processing"
Sep 05 20:04:02.433532 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.433480    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="322b686c-dddc-4880-992f-afa906a71782" containerName="payload-processing"
Sep 05 20:04:02.433683 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.433591    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="44a510ef-1998-46aa-814f-bbe19bea757c" containerName="payload-pre-processing"
Sep 05 20:04:02.433683 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.433604    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="322b686c-dddc-4880-992f-afa906a71782" containerName="payload-processing"
Sep 05 20:04:02.436536 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.436516    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz"
Sep 05 20:04:02.439299 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.439278    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-0cc31a\""
Sep 05 20:04:02.447671 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.447646    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz"]
Sep 05 20:04:02.480679 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.480641    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k"]
Sep 05 20:04:02.483193 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.483173    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k"
Sep 05 20:04:02.484441 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.484415    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3ec0e0-57b6-4289-910c-6cf10b36102b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz\" (UID: \"dc3ec0e0-57b6-4289-910c-6cf10b36102b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz"
Sep 05 20:04:02.484549 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.484480    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxdk9\" (UniqueName: \"kubernetes.io/projected/dc3ec0e0-57b6-4289-910c-6cf10b36102b-kube-api-access-jxdk9\") pod \"payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz\" (UID: \"dc3ec0e0-57b6-4289-910c-6cf10b36102b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz"
Sep 05 20:04:02.485664 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.485644    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-0cc31a-dockercfg-nrh7l\""
Sep 05 20:04:02.501657 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.501632    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k"]
Sep 05 20:04:02.585086 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.585030    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3ec0e0-57b6-4289-910c-6cf10b36102b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz\" (UID: \"dc3ec0e0-57b6-4289-910c-6cf10b36102b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz"
Sep 05 20:04:02.585316 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.585140    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drhnk\" (UniqueName: \"kubernetes.io/projected/7272109d-9e39-4e5b-ba59-ff514f403aec-kube-api-access-drhnk\") pod \"payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k\" (UID: \"7272109d-9e39-4e5b-ba59-ff514f403aec\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k"
Sep 05 20:04:02.585316 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.585180    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7272109d-9e39-4e5b-ba59-ff514f403aec-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k\" (UID: \"7272109d-9e39-4e5b-ba59-ff514f403aec\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k"
Sep 05 20:04:02.585316 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.585232    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxdk9\" (UniqueName: \"kubernetes.io/projected/dc3ec0e0-57b6-4289-910c-6cf10b36102b-kube-api-access-jxdk9\") pod \"payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz\" (UID: \"dc3ec0e0-57b6-4289-910c-6cf10b36102b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz"
Sep 05 20:04:02.586002 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.585971    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3ec0e0-57b6-4289-910c-6cf10b36102b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz\" (UID: \"dc3ec0e0-57b6-4289-910c-6cf10b36102b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz"
Sep 05 20:04:02.594864 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.594823    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxdk9\" (UniqueName: \"kubernetes.io/projected/dc3ec0e0-57b6-4289-910c-6cf10b36102b-kube-api-access-jxdk9\") pod \"payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz\" (UID: \"dc3ec0e0-57b6-4289-910c-6cf10b36102b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz"
Sep 05 20:04:02.686615 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.686515    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drhnk\" (UniqueName: \"kubernetes.io/projected/7272109d-9e39-4e5b-ba59-ff514f403aec-kube-api-access-drhnk\") pod \"payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k\" (UID: \"7272109d-9e39-4e5b-ba59-ff514f403aec\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k"
Sep 05 20:04:02.686615 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.686570    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7272109d-9e39-4e5b-ba59-ff514f403aec-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k\" (UID: \"7272109d-9e39-4e5b-ba59-ff514f403aec\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k"
Sep 05 20:04:02.687373 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.687344    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7272109d-9e39-4e5b-ba59-ff514f403aec-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k\" (UID: \"7272109d-9e39-4e5b-ba59-ff514f403aec\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k"
Sep 05 20:04:02.695810 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.695782    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drhnk\" (UniqueName: \"kubernetes.io/projected/7272109d-9e39-4e5b-ba59-ff514f403aec-kube-api-access-drhnk\") pod \"payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k\" (UID: \"7272109d-9e39-4e5b-ba59-ff514f403aec\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k"
Sep 05 20:04:02.747176 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.747141    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz"
Sep 05 20:04:02.793652 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:02.793616    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k"
Sep 05 20:04:03.027859 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:03.027830    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz"]
Sep 05 20:04:03.029009 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:04:03.028990    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3ec0e0_57b6_4289_910c_6cf10b36102b.slice/crio-2946cf480bdeadddc5e6c538ac213f8115f08b6f8cd15d44bc8c1f1ac97b1959 WatchSource:0}: Error finding container 2946cf480bdeadddc5e6c538ac213f8115f08b6f8cd15d44bc8c1f1ac97b1959: Status 404 returned error can't find the container with id 2946cf480bdeadddc5e6c538ac213f8115f08b6f8cd15d44bc8c1f1ac97b1959
Sep 05 20:04:03.088887 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:03.088855    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k"]
Sep 05 20:04:03.090443 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:04:03.090413    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7272109d_9e39_4e5b_ba59_ff514f403aec.slice/crio-9b34bf59c11d264bb4d90fc52baff27edb38a69ebac19578e1351543f49ec365 WatchSource:0}: Error finding container 9b34bf59c11d264bb4d90fc52baff27edb38a69ebac19578e1351543f49ec365: Status 404 returned error can't find the container with id 9b34bf59c11d264bb4d90fc52baff27edb38a69ebac19578e1351543f49ec365
Sep 05 20:04:03.243179 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:03.243135    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz" event={"ID":"dc3ec0e0-57b6-4289-910c-6cf10b36102b","Type":"ContainerStarted","Data":"4c3df131b361c31ad4482abaa95e02dbe354171c34ab951b24d28547edf75f2b"}
Sep 05 20:04:03.243370 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:03.243208    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz"
Sep 05 20:04:03.243370 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:03.243234    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz" event={"ID":"dc3ec0e0-57b6-4289-910c-6cf10b36102b","Type":"ContainerStarted","Data":"2946cf480bdeadddc5e6c538ac213f8115f08b6f8cd15d44bc8c1f1ac97b1959"}
Sep 05 20:04:03.244966 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:03.244935    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k" event={"ID":"7272109d-9e39-4e5b-ba59-ff514f403aec","Type":"ContainerStarted","Data":"9925c6014c39219963e56d6239a91f45b5709e593c7c56a633ebfaf2539be242"}
Sep 05 20:04:03.245096 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:03.244992    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k"
Sep 05 20:04:03.245096 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:03.245010    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k" event={"ID":"7272109d-9e39-4e5b-ba59-ff514f403aec","Type":"ContainerStarted","Data":"9b34bf59c11d264bb4d90fc52baff27edb38a69ebac19578e1351543f49ec365"}
Sep 05 20:04:03.261537 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:03.261485    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz" podStartSLOduration=1.2614689079999999 podStartE2EDuration="1.261468908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:04:03.2596166 +0000 UTC m=+1242.289248506" watchObservedRunningTime="2026-09-05 20:04:03.261468908 +0000 UTC m=+1242.291100814"
Sep 05 20:04:03.279994 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:03.279881    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k" podStartSLOduration=1.27985857 podStartE2EDuration="1.27985857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:04:03.277153188 +0000 UTC m=+1242.306785097" watchObservedRunningTime="2026-09-05 20:04:03.27985857 +0000 UTC m=+1242.309490482"
Sep 05 20:04:11.922165 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:11.922127    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"]
Sep 05 20:04:14.253561 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:14.253482    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz"
Sep 05 20:04:14.253997 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:14.253729    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k"
Sep 05 20:04:25.908980 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:25.908944    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj"]
Sep 05 20:04:25.914348 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:25.914327    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj"
Sep 05 20:04:25.916998 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:25.916970    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-1436b9-dockercfg-qpn28\""
Sep 05 20:04:25.916998 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:25.916992    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-1436b9\""
Sep 05 20:04:25.923961 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:25.923936    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj"]
Sep 05 20:04:26.007982 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:26.007945    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa28148-f4e9-488f-89fd-4cf9051c538b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj\" (UID: \"6fa28148-f4e9-488f-89fd-4cf9051c538b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj"
Sep 05 20:04:26.008176 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:26.008011    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq7j2\" (UniqueName: \"kubernetes.io/projected/6fa28148-f4e9-488f-89fd-4cf9051c538b-kube-api-access-mq7j2\") pod \"payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj\" (UID: \"6fa28148-f4e9-488f-89fd-4cf9051c538b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj"
Sep 05 20:04:26.108621 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:26.108578    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq7j2\" (UniqueName: \"kubernetes.io/projected/6fa28148-f4e9-488f-89fd-4cf9051c538b-kube-api-access-mq7j2\") pod \"payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj\" (UID: \"6fa28148-f4e9-488f-89fd-4cf9051c538b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj"
Sep 05 20:04:26.108789 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:26.108695    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa28148-f4e9-488f-89fd-4cf9051c538b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj\" (UID: \"6fa28148-f4e9-488f-89fd-4cf9051c538b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj"
Sep 05 20:04:26.109396 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:26.109375    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa28148-f4e9-488f-89fd-4cf9051c538b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj\" (UID: \"6fa28148-f4e9-488f-89fd-4cf9051c538b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj"
Sep 05 20:04:26.116640 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:26.116605    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq7j2\" (UniqueName: \"kubernetes.io/projected/6fa28148-f4e9-488f-89fd-4cf9051c538b-kube-api-access-mq7j2\") pod \"payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj\" (UID: \"6fa28148-f4e9-488f-89fd-4cf9051c538b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj"
Sep 05 20:04:26.226882 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:26.226799    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj"
Sep 05 20:04:26.504295 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:26.504265    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj"]
Sep 05 20:04:26.505216 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:04:26.505189    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa28148_f4e9_488f_89fd_4cf9051c538b.slice/crio-1bc99a066d5e7d2e5f17313dc320554773c1d7a123032e3bed1567be7da281c2 WatchSource:0}: Error finding container 1bc99a066d5e7d2e5f17313dc320554773c1d7a123032e3bed1567be7da281c2: Status 404 returned error can't find the container with id 1bc99a066d5e7d2e5f17313dc320554773c1d7a123032e3bed1567be7da281c2
Sep 05 20:04:26.942828 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:26.942788    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq" podUID="d725dd23-0c45-4bcf-951c-39883f21a95d" containerName="main" containerID="cri-o://e7ae6a59aa38581e3fe7e282de4b8a5861b4c44161d6f1f680ce4c5c14a0ab40" gracePeriod=45
Sep 05 20:04:27.213081 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.213050    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:04:27.319569 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.319535    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d725dd23-0c45-4bcf-951c-39883f21a95d-tmp-dir\") pod \"d725dd23-0c45-4bcf-951c-39883f21a95d\" (UID: \"d725dd23-0c45-4bcf-951c-39883f21a95d\") "
Sep 05 20:04:27.319751 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.319626    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brnnf\" (UniqueName: \"kubernetes.io/projected/d725dd23-0c45-4bcf-951c-39883f21a95d-kube-api-access-brnnf\") pod \"d725dd23-0c45-4bcf-951c-39883f21a95d\" (UID: \"d725dd23-0c45-4bcf-951c-39883f21a95d\") "
Sep 05 20:04:27.319751 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.319644    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d725dd23-0c45-4bcf-951c-39883f21a95d-home\") pod \"d725dd23-0c45-4bcf-951c-39883f21a95d\" (UID: \"d725dd23-0c45-4bcf-951c-39883f21a95d\") "
Sep 05 20:04:27.319751 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.319670    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d725dd23-0c45-4bcf-951c-39883f21a95d-model-cache\") pod \"d725dd23-0c45-4bcf-951c-39883f21a95d\" (UID: \"d725dd23-0c45-4bcf-951c-39883f21a95d\") "
Sep 05 20:04:27.319751 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.319718    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d725dd23-0c45-4bcf-951c-39883f21a95d-tls-certs\") pod \"d725dd23-0c45-4bcf-951c-39883f21a95d\" (UID: \"d725dd23-0c45-4bcf-951c-39883f21a95d\") "
Sep 05 20:04:27.319751 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.319748    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d725dd23-0c45-4bcf-951c-39883f21a95d-dshm\") pod \"d725dd23-0c45-4bcf-951c-39883f21a95d\" (UID: \"d725dd23-0c45-4bcf-951c-39883f21a95d\") "
Sep 05 20:04:27.320035 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.319826    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d725dd23-0c45-4bcf-951c-39883f21a95d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d725dd23-0c45-4bcf-951c-39883f21a95d" (UID: "d725dd23-0c45-4bcf-951c-39883f21a95d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:04:27.320035 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.319922    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d725dd23-0c45-4bcf-951c-39883f21a95d-home" (OuterVolumeSpecName: "home") pod "d725dd23-0c45-4bcf-951c-39883f21a95d" (UID: "d725dd23-0c45-4bcf-951c-39883f21a95d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:04:27.320035 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.319937    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d725dd23-0c45-4bcf-951c-39883f21a95d-model-cache" (OuterVolumeSpecName: "model-cache") pod "d725dd23-0c45-4bcf-951c-39883f21a95d" (UID: "d725dd23-0c45-4bcf-951c-39883f21a95d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:04:27.320214 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.320063    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d725dd23-0c45-4bcf-951c-39883f21a95d-tmp-dir\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:04:27.320214 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.320082    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d725dd23-0c45-4bcf-951c-39883f21a95d-home\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:04:27.320214 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.320096    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d725dd23-0c45-4bcf-951c-39883f21a95d-model-cache\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:04:27.322000 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.321974    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d725dd23-0c45-4bcf-951c-39883f21a95d-kube-api-access-brnnf" (OuterVolumeSpecName: "kube-api-access-brnnf") pod "d725dd23-0c45-4bcf-951c-39883f21a95d" (UID: "d725dd23-0c45-4bcf-951c-39883f21a95d"). InnerVolumeSpecName "kube-api-access-brnnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:04:27.322122 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.321996    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d725dd23-0c45-4bcf-951c-39883f21a95d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d725dd23-0c45-4bcf-951c-39883f21a95d" (UID: "d725dd23-0c45-4bcf-951c-39883f21a95d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:04:27.322122 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.321996    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d725dd23-0c45-4bcf-951c-39883f21a95d-dshm" (OuterVolumeSpecName: "dshm") pod "d725dd23-0c45-4bcf-951c-39883f21a95d" (UID: "d725dd23-0c45-4bcf-951c-39883f21a95d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:04:27.409758 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.409722    2583 generic.go:356] "Generic (PLEG): container finished" podID="d725dd23-0c45-4bcf-951c-39883f21a95d" containerID="e7ae6a59aa38581e3fe7e282de4b8a5861b4c44161d6f1f680ce4c5c14a0ab40" exitCode=0
Sep 05 20:04:27.409945 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.409767    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"
Sep 05 20:04:27.409945 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.409803    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq" event={"ID":"d725dd23-0c45-4bcf-951c-39883f21a95d","Type":"ContainerDied","Data":"e7ae6a59aa38581e3fe7e282de4b8a5861b4c44161d6f1f680ce4c5c14a0ab40"}
Sep 05 20:04:27.409945 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.409847    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq" event={"ID":"d725dd23-0c45-4bcf-951c-39883f21a95d","Type":"ContainerDied","Data":"d23175e62bbb77fabec341e558e75fbaf32617224457e509843618a964e9642d"}
Sep 05 20:04:27.409945 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.409865    2583 scope.go:117] "RemoveContainer" containerID="e7ae6a59aa38581e3fe7e282de4b8a5861b4c44161d6f1f680ce4c5c14a0ab40"
Sep 05 20:04:27.411831 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.411807    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj" event={"ID":"6fa28148-f4e9-488f-89fd-4cf9051c538b","Type":"ContainerStarted","Data":"a48509fd04e5c65e42c11bcabc0ae2d6cda5fdb270bdea8ac485d6d96b5bfd68"}
Sep 05 20:04:27.411957 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.411882    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj" event={"ID":"6fa28148-f4e9-488f-89fd-4cf9051c538b","Type":"ContainerStarted","Data":"1bc99a066d5e7d2e5f17313dc320554773c1d7a123032e3bed1567be7da281c2"}
Sep 05 20:04:27.411957 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.411919    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj"
Sep 05 20:04:27.420848 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.420747    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brnnf\" (UniqueName: \"kubernetes.io/projected/d725dd23-0c45-4bcf-951c-39883f21a95d-kube-api-access-brnnf\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:04:27.420848 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.420768    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d725dd23-0c45-4bcf-951c-39883f21a95d-tls-certs\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:04:27.420848 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.420778    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d725dd23-0c45-4bcf-951c-39883f21a95d-dshm\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:04:27.420996 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.420859    2583 scope.go:117] "RemoveContainer" containerID="e7ae6a59aa38581e3fe7e282de4b8a5861b4c44161d6f1f680ce4c5c14a0ab40"
Sep 05 20:04:27.421188 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:04:27.421170    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7ae6a59aa38581e3fe7e282de4b8a5861b4c44161d6f1f680ce4c5c14a0ab40\": container with ID starting with e7ae6a59aa38581e3fe7e282de4b8a5861b4c44161d6f1f680ce4c5c14a0ab40 not found: ID does not exist" containerID="e7ae6a59aa38581e3fe7e282de4b8a5861b4c44161d6f1f680ce4c5c14a0ab40"
Sep 05 20:04:27.421255 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.421201    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7ae6a59aa38581e3fe7e282de4b8a5861b4c44161d6f1f680ce4c5c14a0ab40"} err="failed to get container status \"e7ae6a59aa38581e3fe7e282de4b8a5861b4c44161d6f1f680ce4c5c14a0ab40\": rpc error: code = NotFound desc = could not find container \"e7ae6a59aa38581e3fe7e282de4b8a5861b4c44161d6f1f680ce4c5c14a0ab40\": container with ID starting with e7ae6a59aa38581e3fe7e282de4b8a5861b4c44161d6f1f680ce4c5c14a0ab40 not found: ID does not exist"
Sep 05 20:04:27.429633 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.429582    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj" podStartSLOduration=2.429564993 podStartE2EDuration="2.429564993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:04:27.427546245 +0000 UTC m=+1266.457178152" watchObservedRunningTime="2026-09-05 20:04:27.429564993 +0000 UTC m=+1266.459196899"
Sep 05 20:04:27.441632 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.441602    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"]
Sep 05 20:04:27.445728 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:27.445706    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq"]
Sep 05 20:04:29.435938 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:29.435900    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d725dd23-0c45-4bcf-951c-39883f21a95d" path="/var/lib/kubelet/pods/d725dd23-0c45-4bcf-951c-39883f21a95d/volumes"
Sep 05 20:04:32.199031 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:32.198976    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-noten-0741a0/noten-model-0741a0-kserve-7c54c56cbc-v6fhq" podUID="d725dd23-0c45-4bcf-951c-39883f21a95d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded"
Sep 05 20:04:38.421250 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:38.421219    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj"
Sep 05 20:04:43.425537 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.425492    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"]
Sep 05 20:04:43.426160 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.426134    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d725dd23-0c45-4bcf-951c-39883f21a95d" containerName="main"
Sep 05 20:04:43.426160 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.426163    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d725dd23-0c45-4bcf-951c-39883f21a95d" containerName="main"
Sep 05 20:04:43.426321 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.426302    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d725dd23-0c45-4bcf-951c-39883f21a95d" containerName="main"
Sep 05 20:04:43.430940 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.430912    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:04:43.433285 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.433265    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4d1885\"/\"default-dockercfg-hrt2t\""
Sep 05 20:04:43.433402 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.433266    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-4d1885\"/\"openshift-service-ca.crt\""
Sep 05 20:04:43.433714 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.433694    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-4d1885\"/\"kube-root-ca.crt\""
Sep 05 20:04:43.433934 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.433725    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4d1885\"/\"auth-test-model-4d1885-kserve-self-signed-certs\""
Sep 05 20:04:43.441205 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.441180    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"]
Sep 05 20:04:43.473416 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.473387    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqj6r\" (UniqueName: \"kubernetes.io/projected/27a73d41-6dea-4e61-bb71-bc3263eb4027-kube-api-access-tqj6r\") pod \"auth-test-model-4d1885-kserve-889d7bdb7-hmn7w\" (UID: \"27a73d41-6dea-4e61-bb71-bc3263eb4027\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:04:43.473580 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.473460    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27a73d41-6dea-4e61-bb71-bc3263eb4027-model-cache\") pod \"auth-test-model-4d1885-kserve-889d7bdb7-hmn7w\" (UID: \"27a73d41-6dea-4e61-bb71-bc3263eb4027\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:04:43.473580 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.473490    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27a73d41-6dea-4e61-bb71-bc3263eb4027-tls-certs\") pod \"auth-test-model-4d1885-kserve-889d7bdb7-hmn7w\" (UID: \"27a73d41-6dea-4e61-bb71-bc3263eb4027\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:04:43.473580 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.473507    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27a73d41-6dea-4e61-bb71-bc3263eb4027-dshm\") pod \"auth-test-model-4d1885-kserve-889d7bdb7-hmn7w\" (UID: \"27a73d41-6dea-4e61-bb71-bc3263eb4027\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:04:43.473761 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.473602    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27a73d41-6dea-4e61-bb71-bc3263eb4027-home\") pod \"auth-test-model-4d1885-kserve-889d7bdb7-hmn7w\" (UID: \"27a73d41-6dea-4e61-bb71-bc3263eb4027\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:04:43.473761 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.473634    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27a73d41-6dea-4e61-bb71-bc3263eb4027-tmp-dir\") pod \"auth-test-model-4d1885-kserve-889d7bdb7-hmn7w\" (UID: \"27a73d41-6dea-4e61-bb71-bc3263eb4027\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:04:43.574815 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.574779    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27a73d41-6dea-4e61-bb71-bc3263eb4027-model-cache\") pod \"auth-test-model-4d1885-kserve-889d7bdb7-hmn7w\" (UID: \"27a73d41-6dea-4e61-bb71-bc3263eb4027\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:04:43.575011 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.574823    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27a73d41-6dea-4e61-bb71-bc3263eb4027-tls-certs\") pod \"auth-test-model-4d1885-kserve-889d7bdb7-hmn7w\" (UID: \"27a73d41-6dea-4e61-bb71-bc3263eb4027\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:04:43.575011 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.574849    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27a73d41-6dea-4e61-bb71-bc3263eb4027-dshm\") pod \"auth-test-model-4d1885-kserve-889d7bdb7-hmn7w\" (UID: \"27a73d41-6dea-4e61-bb71-bc3263eb4027\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:04:43.575011 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.574927    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27a73d41-6dea-4e61-bb71-bc3263eb4027-home\") pod \"auth-test-model-4d1885-kserve-889d7bdb7-hmn7w\" (UID: \"27a73d41-6dea-4e61-bb71-bc3263eb4027\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:04:43.575011 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.574956    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27a73d41-6dea-4e61-bb71-bc3263eb4027-tmp-dir\") pod \"auth-test-model-4d1885-kserve-889d7bdb7-hmn7w\" (UID: \"27a73d41-6dea-4e61-bb71-bc3263eb4027\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:04:43.575271 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.575022    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqj6r\" (UniqueName: \"kubernetes.io/projected/27a73d41-6dea-4e61-bb71-bc3263eb4027-kube-api-access-tqj6r\") pod \"auth-test-model-4d1885-kserve-889d7bdb7-hmn7w\" (UID: \"27a73d41-6dea-4e61-bb71-bc3263eb4027\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:04:43.575331 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.575278    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27a73d41-6dea-4e61-bb71-bc3263eb4027-model-cache\") pod \"auth-test-model-4d1885-kserve-889d7bdb7-hmn7w\" (UID: \"27a73d41-6dea-4e61-bb71-bc3263eb4027\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:04:43.575331 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.575317    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27a73d41-6dea-4e61-bb71-bc3263eb4027-home\") pod \"auth-test-model-4d1885-kserve-889d7bdb7-hmn7w\" (UID: \"27a73d41-6dea-4e61-bb71-bc3263eb4027\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:04:43.575535 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.575512    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27a73d41-6dea-4e61-bb71-bc3263eb4027-tmp-dir\") pod \"auth-test-model-4d1885-kserve-889d7bdb7-hmn7w\" (UID: \"27a73d41-6dea-4e61-bb71-bc3263eb4027\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:04:43.577322 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.577304    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27a73d41-6dea-4e61-bb71-bc3263eb4027-dshm\") pod \"auth-test-model-4d1885-kserve-889d7bdb7-hmn7w\" (UID: \"27a73d41-6dea-4e61-bb71-bc3263eb4027\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:04:43.577452 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.577434    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27a73d41-6dea-4e61-bb71-bc3263eb4027-tls-certs\") pod \"auth-test-model-4d1885-kserve-889d7bdb7-hmn7w\" (UID: \"27a73d41-6dea-4e61-bb71-bc3263eb4027\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:04:43.595246 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.595198    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqj6r\" (UniqueName: \"kubernetes.io/projected/27a73d41-6dea-4e61-bb71-bc3263eb4027-kube-api-access-tqj6r\") pod \"auth-test-model-4d1885-kserve-889d7bdb7-hmn7w\" (UID: \"27a73d41-6dea-4e61-bb71-bc3263eb4027\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:04:43.743141 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:43.743012    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:04:44.026216 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:44.026192    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"]
Sep 05 20:04:44.027753 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:04:44.027724    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27a73d41_6dea_4e61_bb71_bc3263eb4027.slice/crio-9f6f0c69d2159b10e459469f6ae5d28acdc4349a12502cf9d0aac6235a4281ff WatchSource:0}: Error finding container 9f6f0c69d2159b10e459469f6ae5d28acdc4349a12502cf9d0aac6235a4281ff: Status 404 returned error can't find the container with id 9f6f0c69d2159b10e459469f6ae5d28acdc4349a12502cf9d0aac6235a4281ff
Sep 05 20:04:44.521141 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:44.521089    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w" event={"ID":"27a73d41-6dea-4e61-bb71-bc3263eb4027","Type":"ContainerStarted","Data":"414fcb79533b912cebc07a1bbb75c842834a896dc453e1d960ef140208fcbc77"}
Sep 05 20:04:44.521141 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:44.521142    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w" event={"ID":"27a73d41-6dea-4e61-bb71-bc3263eb4027","Type":"ContainerStarted","Data":"9f6f0c69d2159b10e459469f6ae5d28acdc4349a12502cf9d0aac6235a4281ff"}
Sep 05 20:04:44.538961 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:44.538908    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w" podStartSLOduration=1.538892595 podStartE2EDuration="1.538892595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:04:44.537456703 +0000 UTC m=+1283.567088614" watchObservedRunningTime="2026-09-05 20:04:44.538892595 +0000 UTC m=+1283.568524501"
Sep 05 20:04:53.709740 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:53.709705    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4"]
Sep 05 20:04:53.713798 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:53.713774    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4"
Sep 05 20:04:53.716029 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:53.716004    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-ca4fba\""
Sep 05 20:04:53.725434 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:53.725406    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4"]
Sep 05 20:04:53.743743 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:53.743712    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:04:53.743743 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:53.743752    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:04:53.756258 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:53.756233    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:04:53.769923 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:53.769888    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8eb8a4e-6700-4b55-8777-00752b3695b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4\" (UID: \"d8eb8a4e-6700-4b55-8777-00752b3695b9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4"
Sep 05 20:04:53.770079 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:53.769943    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpr6p\" (UniqueName: \"kubernetes.io/projected/d8eb8a4e-6700-4b55-8777-00752b3695b9-kube-api-access-dpr6p\") pod \"payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4\" (UID: \"d8eb8a4e-6700-4b55-8777-00752b3695b9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4"
Sep 05 20:04:53.871350 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:53.871312    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8eb8a4e-6700-4b55-8777-00752b3695b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4\" (UID: \"d8eb8a4e-6700-4b55-8777-00752b3695b9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4"
Sep 05 20:04:53.871561 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:53.871372    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpr6p\" (UniqueName: \"kubernetes.io/projected/d8eb8a4e-6700-4b55-8777-00752b3695b9-kube-api-access-dpr6p\") pod \"payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4\" (UID: \"d8eb8a4e-6700-4b55-8777-00752b3695b9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4"
Sep 05 20:04:53.872078 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:53.872052    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8eb8a4e-6700-4b55-8777-00752b3695b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4\" (UID: \"d8eb8a4e-6700-4b55-8777-00752b3695b9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4"
Sep 05 20:04:53.881898 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:53.881867    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpr6p\" (UniqueName: \"kubernetes.io/projected/d8eb8a4e-6700-4b55-8777-00752b3695b9-kube-api-access-dpr6p\") pod \"payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4\" (UID: \"d8eb8a4e-6700-4b55-8777-00752b3695b9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4"
Sep 05 20:04:54.027155 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:54.027051    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4"
Sep 05 20:04:54.304059 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:54.304029    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4"]
Sep 05 20:04:54.305353 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:04:54.305329    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8eb8a4e_6700_4b55_8777_00752b3695b9.slice/crio-904d094e39f31150ffc12c075cd3012aa2edf095bca55d8df7ceec5e52a9013b WatchSource:0}: Error finding container 904d094e39f31150ffc12c075cd3012aa2edf095bca55d8df7ceec5e52a9013b: Status 404 returned error can't find the container with id 904d094e39f31150ffc12c075cd3012aa2edf095bca55d8df7ceec5e52a9013b
Sep 05 20:04:54.591875 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:54.591838    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4" event={"ID":"d8eb8a4e-6700-4b55-8777-00752b3695b9","Type":"ContainerStarted","Data":"c55b5e31aa1719fdfe96b653aac7bf1195935fa900bf570ee8d74dbbc422bb5b"}
Sep 05 20:04:54.592083 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:54.591897    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4" event={"ID":"d8eb8a4e-6700-4b55-8777-00752b3695b9","Type":"ContainerStarted","Data":"904d094e39f31150ffc12c075cd3012aa2edf095bca55d8df7ceec5e52a9013b"}
Sep 05 20:04:54.592375 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:54.592338    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4"
Sep 05 20:04:54.603319 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:54.603297    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:04:54.608781 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:04:54.608731    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4" podStartSLOduration=1.608714461 podStartE2EDuration="1.608714461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:04:54.607977577 +0000 UTC m=+1293.637609496" watchObservedRunningTime="2026-09-05 20:04:54.608714461 +0000 UTC m=+1293.638346368"
Sep 05 20:05:06.607973 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:06.607941    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4"
Sep 05 20:05:08.631741 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.631705    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"]
Sep 05 20:05:08.636574 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.636546    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:05:08.639405 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.639380    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0cc31a\"/\"auth-test-model-0cc31a-kserve-self-signed-certs\""
Sep 05 20:05:08.639564 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.639428    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0cc31a\"/\"default-dockercfg-g4lbb\""
Sep 05 20:05:08.639564 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.639482    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0cc31a\"/\"openshift-service-ca.crt\""
Sep 05 20:05:08.639564 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.639427    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0cc31a\"/\"kube-root-ca.crt\""
Sep 05 20:05:08.644199 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.643979    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"]
Sep 05 20:05:08.715246 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.715211    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-home\") pod \"auth-test-model-0cc31a-kserve-9c8ddb79-46lqq\" (UID: \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:05:08.715454 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.715282    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-model-cache\") pod \"auth-test-model-0cc31a-kserve-9c8ddb79-46lqq\" (UID: \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:05:08.715454 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.715402    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-tmp-dir\") pod \"auth-test-model-0cc31a-kserve-9c8ddb79-46lqq\" (UID: \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:05:08.715586 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.715460    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-dshm\") pod \"auth-test-model-0cc31a-kserve-9c8ddb79-46lqq\" (UID: \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:05:08.715586 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.715487    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-tls-certs\") pod \"auth-test-model-0cc31a-kserve-9c8ddb79-46lqq\" (UID: \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:05:08.715586 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.715512    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgk48\" (UniqueName: \"kubernetes.io/projected/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-kube-api-access-lgk48\") pod \"auth-test-model-0cc31a-kserve-9c8ddb79-46lqq\" (UID: \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:05:08.816623 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.816587    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-dshm\") pod \"auth-test-model-0cc31a-kserve-9c8ddb79-46lqq\" (UID: \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:05:08.816623 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.816624    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-tls-certs\") pod \"auth-test-model-0cc31a-kserve-9c8ddb79-46lqq\" (UID: \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:05:08.816901 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.816640    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgk48\" (UniqueName: \"kubernetes.io/projected/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-kube-api-access-lgk48\") pod \"auth-test-model-0cc31a-kserve-9c8ddb79-46lqq\" (UID: \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:05:08.816901 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.816697    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-home\") pod \"auth-test-model-0cc31a-kserve-9c8ddb79-46lqq\" (UID: \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:05:08.816901 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.816789    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-model-cache\") pod \"auth-test-model-0cc31a-kserve-9c8ddb79-46lqq\" (UID: \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:05:08.816901 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.816897    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-tmp-dir\") pod \"auth-test-model-0cc31a-kserve-9c8ddb79-46lqq\" (UID: \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:05:08.817104 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.817087    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-home\") pod \"auth-test-model-0cc31a-kserve-9c8ddb79-46lqq\" (UID: \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:05:08.817315 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.817292    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-model-cache\") pod \"auth-test-model-0cc31a-kserve-9c8ddb79-46lqq\" (UID: \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:05:08.817389 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.817337    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-tmp-dir\") pod \"auth-test-model-0cc31a-kserve-9c8ddb79-46lqq\" (UID: \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:05:08.819021 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.818995    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-dshm\") pod \"auth-test-model-0cc31a-kserve-9c8ddb79-46lqq\" (UID: \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:05:08.819172 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.819156    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-tls-certs\") pod \"auth-test-model-0cc31a-kserve-9c8ddb79-46lqq\" (UID: \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:05:08.825247 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.825228    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgk48\" (UniqueName: \"kubernetes.io/projected/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-kube-api-access-lgk48\") pod \"auth-test-model-0cc31a-kserve-9c8ddb79-46lqq\" (UID: \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:05:08.947132 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:08.947033    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:05:09.223416 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:09.223388    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"]
Sep 05 20:05:09.224413 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:05:09.224388    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba1e0e74_26d1_432d_b3c7_0d4926b5fb85.slice/crio-14d2f4c9456427ccefe0e7de2e189dd2bb3e70087f0aae0a25e4b4803e965b38 WatchSource:0}: Error finding container 14d2f4c9456427ccefe0e7de2e189dd2bb3e70087f0aae0a25e4b4803e965b38: Status 404 returned error can't find the container with id 14d2f4c9456427ccefe0e7de2e189dd2bb3e70087f0aae0a25e4b4803e965b38
Sep 05 20:05:09.690536 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:09.690492    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq" event={"ID":"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85","Type":"ContainerStarted","Data":"8783124a3a0686e165e008b3222f90ed7df04aaf77bd9514a1707c8236777e55"}
Sep 05 20:05:09.690927 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:09.690551    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq" event={"ID":"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85","Type":"ContainerStarted","Data":"14d2f4c9456427ccefe0e7de2e189dd2bb3e70087f0aae0a25e4b4803e965b38"}
Sep 05 20:05:09.708952 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:09.708905    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq" podStartSLOduration=1.708890679 podStartE2EDuration="1.708890679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:05:09.708221005 +0000 UTC m=+1308.737852910" watchObservedRunningTime="2026-09-05 20:05:09.708890679 +0000 UTC m=+1308.738522637"
Sep 05 20:05:18.947215 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:18.947174    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:05:18.947642 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:18.947228    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:05:18.959888 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:18.959859    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:05:19.770575 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:19.770541    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:05:21.632495 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:21.632467    2583 scope.go:117] "RemoveContainer" containerID="ea93d86a19bd2d313b24c902a4a0091923beef34a8232a8e1e6f325017378410"
Sep 05 20:05:34.984630 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:34.984585    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg"]
Sep 05 20:05:34.989790 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:34.989765    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg"
Sep 05 20:05:34.991980 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:34.991956    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ada83762\""
Sep 05 20:05:34.992120 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:34.992000    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ada83762-dockercfg-btzrp\""
Sep 05 20:05:35.000279 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:35.000255    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg"]
Sep 05 20:05:35.068742 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:35.068709    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a73cd2b0-39f9-43ad-bff9-ad194f8258b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg\" (UID: \"a73cd2b0-39f9-43ad-bff9-ad194f8258b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg"
Sep 05 20:05:35.068897 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:35.068757    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zlzw\" (UniqueName: \"kubernetes.io/projected/a73cd2b0-39f9-43ad-bff9-ad194f8258b4-kube-api-access-8zlzw\") pod \"payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg\" (UID: \"a73cd2b0-39f9-43ad-bff9-ad194f8258b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg"
Sep 05 20:05:35.169764 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:35.169722    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a73cd2b0-39f9-43ad-bff9-ad194f8258b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg\" (UID: \"a73cd2b0-39f9-43ad-bff9-ad194f8258b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg"
Sep 05 20:05:35.169968 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:35.169782    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zlzw\" (UniqueName: \"kubernetes.io/projected/a73cd2b0-39f9-43ad-bff9-ad194f8258b4-kube-api-access-8zlzw\") pod \"payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg\" (UID: \"a73cd2b0-39f9-43ad-bff9-ad194f8258b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg"
Sep 05 20:05:35.170516 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:35.170493    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a73cd2b0-39f9-43ad-bff9-ad194f8258b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg\" (UID: \"a73cd2b0-39f9-43ad-bff9-ad194f8258b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg"
Sep 05 20:05:35.182856 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:35.182829    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zlzw\" (UniqueName: \"kubernetes.io/projected/a73cd2b0-39f9-43ad-bff9-ad194f8258b4-kube-api-access-8zlzw\") pod \"payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg\" (UID: \"a73cd2b0-39f9-43ad-bff9-ad194f8258b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg"
Sep 05 20:05:35.302499 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:35.302402    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg"
Sep 05 20:05:35.578189 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:35.578159    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg"]
Sep 05 20:05:35.579257 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:05:35.579227    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73cd2b0_39f9_43ad_bff9_ad194f8258b4.slice/crio-a6cd6750598a7eea0cc1697ea66d2049223e0f2133d703a955f42710b6d13ef2 WatchSource:0}: Error finding container a6cd6750598a7eea0cc1697ea66d2049223e0f2133d703a955f42710b6d13ef2: Status 404 returned error can't find the container with id a6cd6750598a7eea0cc1697ea66d2049223e0f2133d703a955f42710b6d13ef2
Sep 05 20:05:35.860041 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:35.859937    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg" event={"ID":"a73cd2b0-39f9-43ad-bff9-ad194f8258b4","Type":"ContainerStarted","Data":"3a3fc3fbb58682df8f50c2b0f920f30d3577858ef5dcdc6bf0ce132b8fe84fe4"}
Sep 05 20:05:35.860041 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:35.860000    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg" event={"ID":"a73cd2b0-39f9-43ad-bff9-ad194f8258b4","Type":"ContainerStarted","Data":"a6cd6750598a7eea0cc1697ea66d2049223e0f2133d703a955f42710b6d13ef2"}
Sep 05 20:05:35.860314 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:35.860056    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg"
Sep 05 20:05:35.879253 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:35.879205    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg" podStartSLOduration=1.8791862350000001 podStartE2EDuration="1.879186235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:05:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:05:35.875881902 +0000 UTC m=+1334.905513809" watchObservedRunningTime="2026-09-05 20:05:35.879186235 +0000 UTC m=+1334.908818142"
Sep 05 20:05:46.874663 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:05:46.874588    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg"
Sep 05 20:06:10.511486 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.511448    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"]
Sep 05 20:06:10.516256 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.516229    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:06:10.519310 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.519138    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ada83762\"/\"default-dockercfg-b59xs\""
Sep 05 20:06:10.519310 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.519178    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ada83762\"/\"openshift-service-ca.crt\""
Sep 05 20:06:10.519310 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.519190    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ada83762\"/\"e2e-lifecycle-model-ada83762-kserve-self-signed-certs\""
Sep 05 20:06:10.519310 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.519138    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ada83762\"/\"kube-root-ca.crt\""
Sep 05 20:06:10.525508 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.524932    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"]
Sep 05 20:06:10.611350 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.611313    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6f79406-d492-4deb-8644-d8df1ab90772-dshm\") pod \"e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7\" (UID: \"b6f79406-d492-4deb-8644-d8df1ab90772\") " pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:06:10.611350 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.611356    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6f79406-d492-4deb-8644-d8df1ab90772-tmp-dir\") pod \"e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7\" (UID: \"b6f79406-d492-4deb-8644-d8df1ab90772\") " pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:06:10.611584 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.611472    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6f79406-d492-4deb-8644-d8df1ab90772-model-cache\") pod \"e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7\" (UID: \"b6f79406-d492-4deb-8644-d8df1ab90772\") " pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:06:10.611584 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.611525    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6f79406-d492-4deb-8644-d8df1ab90772-tls-certs\") pod \"e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7\" (UID: \"b6f79406-d492-4deb-8644-d8df1ab90772\") " pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:06:10.611584 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.611548    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvpkd\" (UniqueName: \"kubernetes.io/projected/b6f79406-d492-4deb-8644-d8df1ab90772-kube-api-access-gvpkd\") pod \"e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7\" (UID: \"b6f79406-d492-4deb-8644-d8df1ab90772\") " pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:06:10.611584 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.611576    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6f79406-d492-4deb-8644-d8df1ab90772-home\") pod \"e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7\" (UID: \"b6f79406-d492-4deb-8644-d8df1ab90772\") " pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:06:10.713045 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.713006    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6f79406-d492-4deb-8644-d8df1ab90772-dshm\") pod \"e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7\" (UID: \"b6f79406-d492-4deb-8644-d8df1ab90772\") " pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:06:10.713249 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.713050    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6f79406-d492-4deb-8644-d8df1ab90772-tmp-dir\") pod \"e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7\" (UID: \"b6f79406-d492-4deb-8644-d8df1ab90772\") " pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:06:10.713249 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.713185    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6f79406-d492-4deb-8644-d8df1ab90772-model-cache\") pod \"e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7\" (UID: \"b6f79406-d492-4deb-8644-d8df1ab90772\") " pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:06:10.713249 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.713233    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6f79406-d492-4deb-8644-d8df1ab90772-tls-certs\") pod \"e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7\" (UID: \"b6f79406-d492-4deb-8644-d8df1ab90772\") " pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:06:10.713376 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.713353    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvpkd\" (UniqueName: \"kubernetes.io/projected/b6f79406-d492-4deb-8644-d8df1ab90772-kube-api-access-gvpkd\") pod \"e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7\" (UID: \"b6f79406-d492-4deb-8644-d8df1ab90772\") " pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:06:10.713436 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.713416    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6f79406-d492-4deb-8644-d8df1ab90772-home\") pod \"e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7\" (UID: \"b6f79406-d492-4deb-8644-d8df1ab90772\") " pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:06:10.713574 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.713553    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6f79406-d492-4deb-8644-d8df1ab90772-tmp-dir\") pod \"e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7\" (UID: \"b6f79406-d492-4deb-8644-d8df1ab90772\") " pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:06:10.713663 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.713601    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6f79406-d492-4deb-8644-d8df1ab90772-model-cache\") pod \"e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7\" (UID: \"b6f79406-d492-4deb-8644-d8df1ab90772\") " pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:06:10.713811 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.713789    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6f79406-d492-4deb-8644-d8df1ab90772-home\") pod \"e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7\" (UID: \"b6f79406-d492-4deb-8644-d8df1ab90772\") " pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:06:10.715361 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.715335    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6f79406-d492-4deb-8644-d8df1ab90772-dshm\") pod \"e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7\" (UID: \"b6f79406-d492-4deb-8644-d8df1ab90772\") " pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:06:10.715764 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.715732    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6f79406-d492-4deb-8644-d8df1ab90772-tls-certs\") pod \"e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7\" (UID: \"b6f79406-d492-4deb-8644-d8df1ab90772\") " pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:06:10.722290 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.722270    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvpkd\" (UniqueName: \"kubernetes.io/projected/b6f79406-d492-4deb-8644-d8df1ab90772-kube-api-access-gvpkd\") pod \"e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7\" (UID: \"b6f79406-d492-4deb-8644-d8df1ab90772\") " pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:06:10.828625 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:10.828598    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:06:11.102257 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:11.102229    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"]
Sep 05 20:06:11.103292 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:06:11.103269    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6f79406_d492_4deb_8644_d8df1ab90772.slice/crio-93a52e72b6fb4a5f17156512877e62bc0c038c07cf003ee05db5dfe6e4ed79ae WatchSource:0}: Error finding container 93a52e72b6fb4a5f17156512877e62bc0c038c07cf003ee05db5dfe6e4ed79ae: Status 404 returned error can't find the container with id 93a52e72b6fb4a5f17156512877e62bc0c038c07cf003ee05db5dfe6e4ed79ae
Sep 05 20:06:12.109091 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:12.109055    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7" event={"ID":"b6f79406-d492-4deb-8644-d8df1ab90772","Type":"ContainerStarted","Data":"ac38ce8ac5d57654958f68f4ac13773be5ed38f3c1ab00146323d0ad02588e94"}
Sep 05 20:06:12.109479 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:12.109121    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7" event={"ID":"b6f79406-d492-4deb-8644-d8df1ab90772","Type":"ContainerStarted","Data":"93a52e72b6fb4a5f17156512877e62bc0c038c07cf003ee05db5dfe6e4ed79ae"}
Sep 05 20:06:12.126769 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:12.126713    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7" podStartSLOduration=2.126697326 podStartE2EDuration="2.126697326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:06:12.125342963 +0000 UTC m=+1371.154974867" watchObservedRunningTime="2026-09-05 20:06:12.126697326 +0000 UTC m=+1371.156329230"
Sep 05 20:06:20.829580 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:20.829535    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:06:20.829580 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:20.829593    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:06:20.842227 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:20.842198    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:06:21.180735 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:21.180641    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:06:21.658952 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:21.658925    2583 scope.go:117] "RemoveContainer" containerID="585974f0690c4bc7badb29e2afdf5b69dc114d5c1ce9c125f610c833d63b6667"
Sep 05 20:06:21.668483 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:21.668460    2583 scope.go:117] "RemoveContainer" containerID="38fdf5855d68371a7c2f298be6aaa2e6c5d7a9a07e9d39e6e1697022308b925e"
Sep 05 20:06:21.677441 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:21.677422    2583 scope.go:117] "RemoveContainer" containerID="38e5434007fe48e999ce81b9473a881b5a254cb7cd320b50427ebd8eb8e58aeb"
Sep 05 20:06:21.686743 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:21.686722    2583 scope.go:117] "RemoveContainer" containerID="14f7e293a49ab60c8e0f3d445885f7405fc03439465da1cc26161c30eba00c5c"
Sep 05 20:06:42.419968 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:42.419899    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ada83762-n9htw"]
Sep 05 20:06:42.425532 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:42.425505    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ada83762-n9htw"
Sep 05 20:06:42.427970 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:42.427943    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 20:06:42.428087 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:42.428014    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zx89\""
Sep 05 20:06:42.432453 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:42.432427    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ada83762-n9htw"]
Sep 05 20:06:42.436899 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:42.436874    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9763df8-630b-44ce-a24c-5f2cce4fbd8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ada83762-n9htw\" (UID: \"f9763df8-630b-44ce-a24c-5f2cce4fbd8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ada83762-n9htw"
Sep 05 20:06:42.537839 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:42.537800    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9763df8-630b-44ce-a24c-5f2cce4fbd8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ada83762-n9htw\" (UID: \"f9763df8-630b-44ce-a24c-5f2cce4fbd8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ada83762-n9htw"
Sep 05 20:06:42.538511 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:42.538489    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9763df8-630b-44ce-a24c-5f2cce4fbd8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ada83762-n9htw\" (UID: \"f9763df8-630b-44ce-a24c-5f2cce4fbd8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ada83762-n9htw"
Sep 05 20:06:42.738596 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:42.738512    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ada83762-n9htw"
Sep 05 20:06:43.023461 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:43.023423    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ada83762-n9htw"]
Sep 05 20:06:43.024364 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:06:43.024320    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9763df8_630b_44ce_a24c_5f2cce4fbd8a.slice/crio-41dcdc6b2f9cd01b4ecac845a38ca255a26495565b928a63393765dffdb1e1a5 WatchSource:0}: Error finding container 41dcdc6b2f9cd01b4ecac845a38ca255a26495565b928a63393765dffdb1e1a5: Status 404 returned error can't find the container with id 41dcdc6b2f9cd01b4ecac845a38ca255a26495565b928a63393765dffdb1e1a5
Sep 05 20:06:43.309888 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:43.309842    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ada83762-n9htw" event={"ID":"f9763df8-630b-44ce-a24c-5f2cce4fbd8a","Type":"ContainerStarted","Data":"41dcdc6b2f9cd01b4ecac845a38ca255a26495565b928a63393765dffdb1e1a5"}
Sep 05 20:06:44.317947 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:44.317913    2583 generic.go:356] "Generic (PLEG): container finished" podID="f9763df8-630b-44ce-a24c-5f2cce4fbd8a" containerID="b0ad4fa5e4967340b4da4c45906e37c489c7a14c410943e4ccd327eadff1ea88" exitCode=0
Sep 05 20:06:44.318368 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:44.317993    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ada83762-n9htw" event={"ID":"f9763df8-630b-44ce-a24c-5f2cce4fbd8a","Type":"ContainerDied","Data":"b0ad4fa5e4967340b4da4c45906e37c489c7a14c410943e4ccd327eadff1ea88"}
Sep 05 20:06:45.463259 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:45.463232    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ada83762-n9htw"
Sep 05 20:06:45.569623 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:45.569582    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9763df8-630b-44ce-a24c-5f2cce4fbd8a-maas-api-service-ca\") pod \"f9763df8-630b-44ce-a24c-5f2cce4fbd8a\" (UID: \"f9763df8-630b-44ce-a24c-5f2cce4fbd8a\") "
Sep 05 20:06:45.569938 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:45.569912    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9763df8-630b-44ce-a24c-5f2cce4fbd8a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f9763df8-630b-44ce-a24c-5f2cce4fbd8a" (UID: "f9763df8-630b-44ce-a24c-5f2cce4fbd8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:06:45.671074 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:45.670993    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9763df8-630b-44ce-a24c-5f2cce4fbd8a-maas-api-service-ca\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:06:46.332560 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:46.332521    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ada83762-n9htw" event={"ID":"f9763df8-630b-44ce-a24c-5f2cce4fbd8a","Type":"ContainerDied","Data":"41dcdc6b2f9cd01b4ecac845a38ca255a26495565b928a63393765dffdb1e1a5"}
Sep 05 20:06:46.332735 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:46.332579    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41dcdc6b2f9cd01b4ecac845a38ca255a26495565b928a63393765dffdb1e1a5"
Sep 05 20:06:46.332735 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:46.332532    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ada83762-n9htw"
Sep 05 20:06:57.969435 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:57.969396    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg"]
Sep 05 20:06:57.970355 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:57.970298    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg" podUID="a73cd2b0-39f9-43ad-bff9-ad194f8258b4" containerName="payload-pre-processing" containerID="cri-o://3a3fc3fbb58682df8f50c2b0f920f30d3577858ef5dcdc6bf0ce132b8fe84fe4" gracePeriod=30
Sep 05 20:06:58.124077 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:58.124032    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ada83762-n9htw"]
Sep 05 20:06:58.126934 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:58.126910    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ada83762-n9htw"]
Sep 05 20:06:58.245957 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:58.245925    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg"
Sep 05 20:06:58.299688 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:58.299649    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a73cd2b0-39f9-43ad-bff9-ad194f8258b4-plugins-config-volume\") pod \"a73cd2b0-39f9-43ad-bff9-ad194f8258b4\" (UID: \"a73cd2b0-39f9-43ad-bff9-ad194f8258b4\") "
Sep 05 20:06:58.299892 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:58.299746    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zlzw\" (UniqueName: \"kubernetes.io/projected/a73cd2b0-39f9-43ad-bff9-ad194f8258b4-kube-api-access-8zlzw\") pod \"a73cd2b0-39f9-43ad-bff9-ad194f8258b4\" (UID: \"a73cd2b0-39f9-43ad-bff9-ad194f8258b4\") "
Sep 05 20:06:58.300083 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:58.300058    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a73cd2b0-39f9-43ad-bff9-ad194f8258b4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a73cd2b0-39f9-43ad-bff9-ad194f8258b4" (UID: "a73cd2b0-39f9-43ad-bff9-ad194f8258b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:06:58.301838 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:58.301812    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a73cd2b0-39f9-43ad-bff9-ad194f8258b4-kube-api-access-8zlzw" (OuterVolumeSpecName: "kube-api-access-8zlzw") pod "a73cd2b0-39f9-43ad-bff9-ad194f8258b4" (UID: "a73cd2b0-39f9-43ad-bff9-ad194f8258b4"). InnerVolumeSpecName "kube-api-access-8zlzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:06:58.401155 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:58.401096    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zlzw\" (UniqueName: \"kubernetes.io/projected/a73cd2b0-39f9-43ad-bff9-ad194f8258b4-kube-api-access-8zlzw\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:06:58.401155 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:58.401143    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a73cd2b0-39f9-43ad-bff9-ad194f8258b4-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:06:58.408964 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:58.408931    2583 generic.go:356] "Generic (PLEG): container finished" podID="a73cd2b0-39f9-43ad-bff9-ad194f8258b4" containerID="3a3fc3fbb58682df8f50c2b0f920f30d3577858ef5dcdc6bf0ce132b8fe84fe4" exitCode=0
Sep 05 20:06:58.409125 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:58.408970    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg" event={"ID":"a73cd2b0-39f9-43ad-bff9-ad194f8258b4","Type":"ContainerDied","Data":"3a3fc3fbb58682df8f50c2b0f920f30d3577858ef5dcdc6bf0ce132b8fe84fe4"}
Sep 05 20:06:58.409125 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:58.409008    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg" event={"ID":"a73cd2b0-39f9-43ad-bff9-ad194f8258b4","Type":"ContainerDied","Data":"a6cd6750598a7eea0cc1697ea66d2049223e0f2133d703a955f42710b6d13ef2"}
Sep 05 20:06:58.409125 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:58.408973    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg"
Sep 05 20:06:58.409125 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:58.409029    2583 scope.go:117] "RemoveContainer" containerID="3a3fc3fbb58682df8f50c2b0f920f30d3577858ef5dcdc6bf0ce132b8fe84fe4"
Sep 05 20:06:58.418634 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:58.418612    2583 scope.go:117] "RemoveContainer" containerID="3a3fc3fbb58682df8f50c2b0f920f30d3577858ef5dcdc6bf0ce132b8fe84fe4"
Sep 05 20:06:58.418901 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:06:58.418878    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a3fc3fbb58682df8f50c2b0f920f30d3577858ef5dcdc6bf0ce132b8fe84fe4\": container with ID starting with 3a3fc3fbb58682df8f50c2b0f920f30d3577858ef5dcdc6bf0ce132b8fe84fe4 not found: ID does not exist" containerID="3a3fc3fbb58682df8f50c2b0f920f30d3577858ef5dcdc6bf0ce132b8fe84fe4"
Sep 05 20:06:58.418974 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:58.418910    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a3fc3fbb58682df8f50c2b0f920f30d3577858ef5dcdc6bf0ce132b8fe84fe4"} err="failed to get container status \"3a3fc3fbb58682df8f50c2b0f920f30d3577858ef5dcdc6bf0ce132b8fe84fe4\": rpc error: code = NotFound desc = could not find container \"3a3fc3fbb58682df8f50c2b0f920f30d3577858ef5dcdc6bf0ce132b8fe84fe4\": container with ID starting with 3a3fc3fbb58682df8f50c2b0f920f30d3577858ef5dcdc6bf0ce132b8fe84fe4 not found: ID does not exist"
Sep 05 20:06:58.430700 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:58.430672    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg"]
Sep 05 20:06:58.435541 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:58.435518    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ada83762-76bd5cb586-t4hgg"]
Sep 05 20:06:59.435771 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:59.435733    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a73cd2b0-39f9-43ad-bff9-ad194f8258b4" path="/var/lib/kubelet/pods/a73cd2b0-39f9-43ad-bff9-ad194f8258b4/volumes"
Sep 05 20:06:59.436177 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:06:59.436091    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9763df8-630b-44ce-a24c-5f2cce4fbd8a" path="/var/lib/kubelet/pods/f9763df8-630b-44ce-a24c-5f2cce4fbd8a/volumes"
Sep 05 20:07:06.090604 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:06.090561    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"]
Sep 05 20:07:15.312453 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:15.312103    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"]
Sep 05 20:07:16.503149 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:16.503091    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"]
Sep 05 20:07:21.111169 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.111129    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7" podUID="b6f79406-d492-4deb-8644-d8df1ab90772" containerName="main" containerID="cri-o://ac38ce8ac5d57654958f68f4ac13773be5ed38f3c1ab00146323d0ad02588e94" gracePeriod=45
Sep 05 20:07:21.167897 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.167853    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7" podUID="b6f79406-d492-4deb-8644-d8df1ab90772" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": dial tcp 10.134.0.90:8000: connect: connection refused"
Sep 05 20:07:21.374521 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.374452    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:07:21.428613 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.428578    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6f79406-d492-4deb-8644-d8df1ab90772-dshm\") pod \"b6f79406-d492-4deb-8644-d8df1ab90772\" (UID: \"b6f79406-d492-4deb-8644-d8df1ab90772\") "
Sep 05 20:07:21.428803 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.428629    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6f79406-d492-4deb-8644-d8df1ab90772-home\") pod \"b6f79406-d492-4deb-8644-d8df1ab90772\" (UID: \"b6f79406-d492-4deb-8644-d8df1ab90772\") "
Sep 05 20:07:21.428803 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.428679    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvpkd\" (UniqueName: \"kubernetes.io/projected/b6f79406-d492-4deb-8644-d8df1ab90772-kube-api-access-gvpkd\") pod \"b6f79406-d492-4deb-8644-d8df1ab90772\" (UID: \"b6f79406-d492-4deb-8644-d8df1ab90772\") "
Sep 05 20:07:21.428803 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.428719    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6f79406-d492-4deb-8644-d8df1ab90772-tls-certs\") pod \"b6f79406-d492-4deb-8644-d8df1ab90772\" (UID: \"b6f79406-d492-4deb-8644-d8df1ab90772\") "
Sep 05 20:07:21.428803 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.428754    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6f79406-d492-4deb-8644-d8df1ab90772-model-cache\") pod \"b6f79406-d492-4deb-8644-d8df1ab90772\" (UID: \"b6f79406-d492-4deb-8644-d8df1ab90772\") "
Sep 05 20:07:21.429139 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.428823    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6f79406-d492-4deb-8644-d8df1ab90772-tmp-dir\") pod \"b6f79406-d492-4deb-8644-d8df1ab90772\" (UID: \"b6f79406-d492-4deb-8644-d8df1ab90772\") "
Sep 05 20:07:21.429139 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.428958    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f79406-d492-4deb-8644-d8df1ab90772-home" (OuterVolumeSpecName: "home") pod "b6f79406-d492-4deb-8644-d8df1ab90772" (UID: "b6f79406-d492-4deb-8644-d8df1ab90772"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:07:21.429265 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.429217    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f79406-d492-4deb-8644-d8df1ab90772-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b6f79406-d492-4deb-8644-d8df1ab90772" (UID: "b6f79406-d492-4deb-8644-d8df1ab90772"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:07:21.429323 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.429313    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6f79406-d492-4deb-8644-d8df1ab90772-home\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:07:21.429379 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.429331    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6f79406-d492-4deb-8644-d8df1ab90772-tmp-dir\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:07:21.429434 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.429415    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f79406-d492-4deb-8644-d8df1ab90772-model-cache" (OuterVolumeSpecName: "model-cache") pod "b6f79406-d492-4deb-8644-d8df1ab90772" (UID: "b6f79406-d492-4deb-8644-d8df1ab90772"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:07:21.431870 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.431825    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f79406-d492-4deb-8644-d8df1ab90772-kube-api-access-gvpkd" (OuterVolumeSpecName: "kube-api-access-gvpkd") pod "b6f79406-d492-4deb-8644-d8df1ab90772" (UID: "b6f79406-d492-4deb-8644-d8df1ab90772"). InnerVolumeSpecName "kube-api-access-gvpkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:07:21.432087 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.432058    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f79406-d492-4deb-8644-d8df1ab90772-dshm" (OuterVolumeSpecName: "dshm") pod "b6f79406-d492-4deb-8644-d8df1ab90772" (UID: "b6f79406-d492-4deb-8644-d8df1ab90772"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:07:21.432499 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.432416    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6f79406-d492-4deb-8644-d8df1ab90772-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b6f79406-d492-4deb-8644-d8df1ab90772" (UID: "b6f79406-d492-4deb-8644-d8df1ab90772"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:07:21.529741 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.529705    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6f79406-d492-4deb-8644-d8df1ab90772-model-cache\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:07:21.529741 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.529736    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6f79406-d492-4deb-8644-d8df1ab90772-dshm\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:07:21.529741 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.529745    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvpkd\" (UniqueName: \"kubernetes.io/projected/b6f79406-d492-4deb-8644-d8df1ab90772-kube-api-access-gvpkd\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:07:21.529980 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.529754    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6f79406-d492-4deb-8644-d8df1ab90772-tls-certs\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:07:21.568522 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.568489    2583 generic.go:356] "Generic (PLEG): container finished" podID="b6f79406-d492-4deb-8644-d8df1ab90772" containerID="ac38ce8ac5d57654958f68f4ac13773be5ed38f3c1ab00146323d0ad02588e94" exitCode=0
Sep 05 20:07:21.568719 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.568533    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"
Sep 05 20:07:21.568719 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.568576    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7" event={"ID":"b6f79406-d492-4deb-8644-d8df1ab90772","Type":"ContainerDied","Data":"ac38ce8ac5d57654958f68f4ac13773be5ed38f3c1ab00146323d0ad02588e94"}
Sep 05 20:07:21.568719 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.568626    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7" event={"ID":"b6f79406-d492-4deb-8644-d8df1ab90772","Type":"ContainerDied","Data":"93a52e72b6fb4a5f17156512877e62bc0c038c07cf003ee05db5dfe6e4ed79ae"}
Sep 05 20:07:21.568719 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.568649    2583 scope.go:117] "RemoveContainer" containerID="ac38ce8ac5d57654958f68f4ac13773be5ed38f3c1ab00146323d0ad02588e94"
Sep 05 20:07:21.579265 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.579226    2583 scope.go:117] "RemoveContainer" containerID="ac38ce8ac5d57654958f68f4ac13773be5ed38f3c1ab00146323d0ad02588e94"
Sep 05 20:07:21.579574 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:07:21.579549    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac38ce8ac5d57654958f68f4ac13773be5ed38f3c1ab00146323d0ad02588e94\": container with ID starting with ac38ce8ac5d57654958f68f4ac13773be5ed38f3c1ab00146323d0ad02588e94 not found: ID does not exist" containerID="ac38ce8ac5d57654958f68f4ac13773be5ed38f3c1ab00146323d0ad02588e94"
Sep 05 20:07:21.579677 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.579580    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac38ce8ac5d57654958f68f4ac13773be5ed38f3c1ab00146323d0ad02588e94"} err="failed to get container status \"ac38ce8ac5d57654958f68f4ac13773be5ed38f3c1ab00146323d0ad02588e94\": rpc error: code = NotFound desc = could not find container \"ac38ce8ac5d57654958f68f4ac13773be5ed38f3c1ab00146323d0ad02588e94\": container with ID starting with ac38ce8ac5d57654958f68f4ac13773be5ed38f3c1ab00146323d0ad02588e94 not found: ID does not exist"
Sep 05 20:07:21.585509 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.585481    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"]
Sep 05 20:07:21.589922 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.589897    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-ada83762/e2e-lifecycle-model-ada83762-kserve-66bf9b58c9-s69w7"]
Sep 05 20:07:21.731818 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.731730    2583 scope.go:117] "RemoveContainer" containerID="714ce1318b52a993c4d1e67ef8db3a8cad51831fc620ada52a133189eaa8282f"
Sep 05 20:07:21.740438 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:21.740417    2583 scope.go:117] "RemoveContainer" containerID="870e6386332b6e10ef146e4a9b0908d734115d415cc0b64279e347c09807ea7c"
Sep 05 20:07:23.435261 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:23.435225    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6f79406-d492-4deb-8644-d8df1ab90772" path="/var/lib/kubelet/pods/b6f79406-d492-4deb-8644-d8df1ab90772/volumes"
Sep 05 20:07:29.348608 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.348564    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"]
Sep 05 20:07:29.349072 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.348993    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a73cd2b0-39f9-43ad-bff9-ad194f8258b4" containerName="payload-pre-processing"
Sep 05 20:07:29.349072 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.349009    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a73cd2b0-39f9-43ad-bff9-ad194f8258b4" containerName="payload-pre-processing"
Sep 05 20:07:29.349072 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.349019    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9763df8-630b-44ce-a24c-5f2cce4fbd8a" containerName="revoke-keys"
Sep 05 20:07:29.349072 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.349024    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9763df8-630b-44ce-a24c-5f2cce4fbd8a" containerName="revoke-keys"
Sep 05 20:07:29.349072 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.349044    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6f79406-d492-4deb-8644-d8df1ab90772" containerName="main"
Sep 05 20:07:29.349072 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.349049    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6f79406-d492-4deb-8644-d8df1ab90772" containerName="main"
Sep 05 20:07:29.349454 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.349134    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a73cd2b0-39f9-43ad-bff9-ad194f8258b4" containerName="payload-pre-processing"
Sep 05 20:07:29.349454 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.349146    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6f79406-d492-4deb-8644-d8df1ab90772" containerName="main"
Sep 05 20:07:29.349454 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.349158    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9763df8-630b-44ce-a24c-5f2cce4fbd8a" containerName="revoke-keys"
Sep 05 20:07:29.352394 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.352374    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:07:29.357338 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.357315    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-4d1885\"/\"sub-test-model-4d1885-kserve-self-signed-certs\""
Sep 05 20:07:29.373280 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.373240    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"]
Sep 05 20:07:29.403303 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.403270    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-tls-certs\") pod \"sub-test-model-4d1885-kserve-56969f4d8c-l5ssg\" (UID: \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:07:29.403303 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.403311    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-tmp-dir\") pod \"sub-test-model-4d1885-kserve-56969f4d8c-l5ssg\" (UID: \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:07:29.403554 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.403416    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-home\") pod \"sub-test-model-4d1885-kserve-56969f4d8c-l5ssg\" (UID: \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:07:29.403554 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.403444    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-dshm\") pod \"sub-test-model-4d1885-kserve-56969f4d8c-l5ssg\" (UID: \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:07:29.403672 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.403558    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-model-cache\") pod \"sub-test-model-4d1885-kserve-56969f4d8c-l5ssg\" (UID: \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:07:29.403672 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.403588    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjpzc\" (UniqueName: \"kubernetes.io/projected/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-kube-api-access-tjpzc\") pod \"sub-test-model-4d1885-kserve-56969f4d8c-l5ssg\" (UID: \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:07:29.504582 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.504544    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-tls-certs\") pod \"sub-test-model-4d1885-kserve-56969f4d8c-l5ssg\" (UID: \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:07:29.504582 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.504586    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-tmp-dir\") pod \"sub-test-model-4d1885-kserve-56969f4d8c-l5ssg\" (UID: \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:07:29.504837 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.504631    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-home\") pod \"sub-test-model-4d1885-kserve-56969f4d8c-l5ssg\" (UID: \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:07:29.504837 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.504652    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-dshm\") pod \"sub-test-model-4d1885-kserve-56969f4d8c-l5ssg\" (UID: \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:07:29.504837 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.504700    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-model-cache\") pod \"sub-test-model-4d1885-kserve-56969f4d8c-l5ssg\" (UID: \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:07:29.504837 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.504727    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjpzc\" (UniqueName: \"kubernetes.io/projected/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-kube-api-access-tjpzc\") pod \"sub-test-model-4d1885-kserve-56969f4d8c-l5ssg\" (UID: \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:07:29.505155 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.505093    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-tmp-dir\") pod \"sub-test-model-4d1885-kserve-56969f4d8c-l5ssg\" (UID: \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:07:29.505259 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.505183    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-model-cache\") pod \"sub-test-model-4d1885-kserve-56969f4d8c-l5ssg\" (UID: \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:07:29.505344 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.505327    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-home\") pod \"sub-test-model-4d1885-kserve-56969f4d8c-l5ssg\" (UID: \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:07:29.506927 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.506907    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-dshm\") pod \"sub-test-model-4d1885-kserve-56969f4d8c-l5ssg\" (UID: \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:07:29.507141 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.507088    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-tls-certs\") pod \"sub-test-model-4d1885-kserve-56969f4d8c-l5ssg\" (UID: \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:07:29.514102 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.514074    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjpzc\" (UniqueName: \"kubernetes.io/projected/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-kube-api-access-tjpzc\") pod \"sub-test-model-4d1885-kserve-56969f4d8c-l5ssg\" (UID: \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\") " pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:07:29.662316 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.662218    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:07:29.938362 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:29.938335    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"]
Sep 05 20:07:29.940042 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:07:29.940004    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8aa7ef6_d244_49d2_bfc9_478d8fece05e.slice/crio-be862ed7a9faae4b7b50c9ecdb783ece05c3e329cdcb4085e159aa7e3b739701 WatchSource:0}: Error finding container be862ed7a9faae4b7b50c9ecdb783ece05c3e329cdcb4085e159aa7e3b739701: Status 404 returned error can't find the container with id be862ed7a9faae4b7b50c9ecdb783ece05c3e329cdcb4085e159aa7e3b739701
Sep 05 20:07:30.334027 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.333983    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w" podUID="27a73d41-6dea-4e61-bb71-bc3263eb4027" containerName="main" containerID="cri-o://414fcb79533b912cebc07a1bbb75c842834a896dc453e1d960ef140208fcbc77" gracePeriod=45
Sep 05 20:07:30.597246 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.597178    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:07:30.614812 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.614785    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27a73d41-6dea-4e61-bb71-bc3263eb4027-home\") pod \"27a73d41-6dea-4e61-bb71-bc3263eb4027\" (UID: \"27a73d41-6dea-4e61-bb71-bc3263eb4027\") "
Sep 05 20:07:30.614993 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.614821    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27a73d41-6dea-4e61-bb71-bc3263eb4027-dshm\") pod \"27a73d41-6dea-4e61-bb71-bc3263eb4027\" (UID: \"27a73d41-6dea-4e61-bb71-bc3263eb4027\") "
Sep 05 20:07:30.614993 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.614877    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27a73d41-6dea-4e61-bb71-bc3263eb4027-tmp-dir\") pod \"27a73d41-6dea-4e61-bb71-bc3263eb4027\" (UID: \"27a73d41-6dea-4e61-bb71-bc3263eb4027\") "
Sep 05 20:07:30.614993 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.614909    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27a73d41-6dea-4e61-bb71-bc3263eb4027-tls-certs\") pod \"27a73d41-6dea-4e61-bb71-bc3263eb4027\" (UID: \"27a73d41-6dea-4e61-bb71-bc3263eb4027\") "
Sep 05 20:07:30.615208 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.615004    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27a73d41-6dea-4e61-bb71-bc3263eb4027-model-cache\") pod \"27a73d41-6dea-4e61-bb71-bc3263eb4027\" (UID: \"27a73d41-6dea-4e61-bb71-bc3263eb4027\") "
Sep 05 20:07:30.615208 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.615080    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a73d41-6dea-4e61-bb71-bc3263eb4027-home" (OuterVolumeSpecName: "home") pod "27a73d41-6dea-4e61-bb71-bc3263eb4027" (UID: "27a73d41-6dea-4e61-bb71-bc3263eb4027"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:07:30.615208 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.615101    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqj6r\" (UniqueName: \"kubernetes.io/projected/27a73d41-6dea-4e61-bb71-bc3263eb4027-kube-api-access-tqj6r\") pod \"27a73d41-6dea-4e61-bb71-bc3263eb4027\" (UID: \"27a73d41-6dea-4e61-bb71-bc3263eb4027\") "
Sep 05 20:07:30.615208 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.615164    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a73d41-6dea-4e61-bb71-bc3263eb4027-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "27a73d41-6dea-4e61-bb71-bc3263eb4027" (UID: "27a73d41-6dea-4e61-bb71-bc3263eb4027"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:07:30.615482 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.615294    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a73d41-6dea-4e61-bb71-bc3263eb4027-model-cache" (OuterVolumeSpecName: "model-cache") pod "27a73d41-6dea-4e61-bb71-bc3263eb4027" (UID: "27a73d41-6dea-4e61-bb71-bc3263eb4027"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:07:30.615662 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.615595    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27a73d41-6dea-4e61-bb71-bc3263eb4027-model-cache\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:07:30.615662 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.615619    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27a73d41-6dea-4e61-bb71-bc3263eb4027-home\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:07:30.615662 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.615634    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27a73d41-6dea-4e61-bb71-bc3263eb4027-tmp-dir\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:07:30.618734 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.618613    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a73d41-6dea-4e61-bb71-bc3263eb4027-dshm" (OuterVolumeSpecName: "dshm") pod "27a73d41-6dea-4e61-bb71-bc3263eb4027" (UID: "27a73d41-6dea-4e61-bb71-bc3263eb4027"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:07:30.618870 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.618742    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27a73d41-6dea-4e61-bb71-bc3263eb4027-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27a73d41-6dea-4e61-bb71-bc3263eb4027" (UID: "27a73d41-6dea-4e61-bb71-bc3263eb4027"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:07:30.618870 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.618760    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a73d41-6dea-4e61-bb71-bc3263eb4027-kube-api-access-tqj6r" (OuterVolumeSpecName: "kube-api-access-tqj6r") pod "27a73d41-6dea-4e61-bb71-bc3263eb4027" (UID: "27a73d41-6dea-4e61-bb71-bc3263eb4027"). InnerVolumeSpecName "kube-api-access-tqj6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:07:30.631306 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.631272    2583 generic.go:356] "Generic (PLEG): container finished" podID="27a73d41-6dea-4e61-bb71-bc3263eb4027" containerID="414fcb79533b912cebc07a1bbb75c842834a896dc453e1d960ef140208fcbc77" exitCode=0
Sep 05 20:07:30.631437 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.631312    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"
Sep 05 20:07:30.631437 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.631348    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w" event={"ID":"27a73d41-6dea-4e61-bb71-bc3263eb4027","Type":"ContainerDied","Data":"414fcb79533b912cebc07a1bbb75c842834a896dc453e1d960ef140208fcbc77"}
Sep 05 20:07:30.631437 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.631388    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w" event={"ID":"27a73d41-6dea-4e61-bb71-bc3263eb4027","Type":"ContainerDied","Data":"9f6f0c69d2159b10e459469f6ae5d28acdc4349a12502cf9d0aac6235a4281ff"}
Sep 05 20:07:30.631437 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.631410    2583 scope.go:117] "RemoveContainer" containerID="414fcb79533b912cebc07a1bbb75c842834a896dc453e1d960ef140208fcbc77"
Sep 05 20:07:30.633455 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.633419    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg" event={"ID":"a8aa7ef6-d244-49d2-bfc9-478d8fece05e","Type":"ContainerStarted","Data":"74e4ad3e8978af686047b4725ff14c118197cc071922cb1e70e5ed6b6e4a7791"}
Sep 05 20:07:30.633583 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.633462    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg" event={"ID":"a8aa7ef6-d244-49d2-bfc9-478d8fece05e","Type":"ContainerStarted","Data":"be862ed7a9faae4b7b50c9ecdb783ece05c3e329cdcb4085e159aa7e3b739701"}
Sep 05 20:07:30.642564 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.642545    2583 scope.go:117] "RemoveContainer" containerID="414fcb79533b912cebc07a1bbb75c842834a896dc453e1d960ef140208fcbc77"
Sep 05 20:07:30.642871 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:07:30.642853    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"414fcb79533b912cebc07a1bbb75c842834a896dc453e1d960ef140208fcbc77\": container with ID starting with 414fcb79533b912cebc07a1bbb75c842834a896dc453e1d960ef140208fcbc77 not found: ID does not exist" containerID="414fcb79533b912cebc07a1bbb75c842834a896dc453e1d960ef140208fcbc77"
Sep 05 20:07:30.642927 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.642879    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"414fcb79533b912cebc07a1bbb75c842834a896dc453e1d960ef140208fcbc77"} err="failed to get container status \"414fcb79533b912cebc07a1bbb75c842834a896dc453e1d960ef140208fcbc77\": rpc error: code = NotFound desc = could not find container \"414fcb79533b912cebc07a1bbb75c842834a896dc453e1d960ef140208fcbc77\": container with ID starting with 414fcb79533b912cebc07a1bbb75c842834a896dc453e1d960ef140208fcbc77 not found: ID does not exist"
Sep 05 20:07:30.654704 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.654666    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg" podStartSLOduration=1.65465101 podStartE2EDuration="1.65465101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:07:30.652441995 +0000 UTC m=+1449.682073900" watchObservedRunningTime="2026-09-05 20:07:30.65465101 +0000 UTC m=+1449.684282914"
Sep 05 20:07:30.669098 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.669068    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"]
Sep 05 20:07:30.674919 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.674893    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4d1885/auth-test-model-4d1885-kserve-889d7bdb7-hmn7w"]
Sep 05 20:07:30.716389 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.716346    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqj6r\" (UniqueName: \"kubernetes.io/projected/27a73d41-6dea-4e61-bb71-bc3263eb4027-kube-api-access-tqj6r\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:07:30.716389 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.716383    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27a73d41-6dea-4e61-bb71-bc3263eb4027-dshm\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:07:30.716389 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:30.716393    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27a73d41-6dea-4e61-bb71-bc3263eb4027-tls-certs\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:07:31.435588 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:31.435554    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27a73d41-6dea-4e61-bb71-bc3263eb4027" path="/var/lib/kubelet/pods/27a73d41-6dea-4e61-bb71-bc3263eb4027/volumes"
Sep 05 20:07:31.523283 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:31.523241    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq" podUID="ba1e0e74-26d1-432d-b3c7-0d4926b5fb85" containerName="main" containerID="cri-o://8783124a3a0686e165e008b3222f90ed7df04aaf77bd9514a1707c8236777e55" gracePeriod=45
Sep 05 20:07:31.641780 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:31.641744    2583 generic.go:356] "Generic (PLEG): container finished" podID="ba1e0e74-26d1-432d-b3c7-0d4926b5fb85" containerID="8783124a3a0686e165e008b3222f90ed7df04aaf77bd9514a1707c8236777e55" exitCode=0
Sep 05 20:07:31.642433 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:31.641855    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq" event={"ID":"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85","Type":"ContainerDied","Data":"8783124a3a0686e165e008b3222f90ed7df04aaf77bd9514a1707c8236777e55"}
Sep 05 20:07:31.790857 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:31.790825    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:07:31.827988 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:31.827956    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgk48\" (UniqueName: \"kubernetes.io/projected/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-kube-api-access-lgk48\") pod \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\" (UID: \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\") "
Sep 05 20:07:31.828192 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:31.828016    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-dshm\") pod \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\" (UID: \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\") "
Sep 05 20:07:31.828192 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:31.828086    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-model-cache\") pod \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\" (UID: \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\") "
Sep 05 20:07:31.828192 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:31.828179    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-home\") pod \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\" (UID: \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\") "
Sep 05 20:07:31.828381 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:31.828210    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-tmp-dir\") pod \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\" (UID: \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\") "
Sep 05 20:07:31.828381 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:31.828255    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-tls-certs\") pod \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\" (UID: \"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85\") "
Sep 05 20:07:31.828533 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:31.828488    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-home" (OuterVolumeSpecName: "home") pod "ba1e0e74-26d1-432d-b3c7-0d4926b5fb85" (UID: "ba1e0e74-26d1-432d-b3c7-0d4926b5fb85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:07:31.828533 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:31.828501    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba1e0e74-26d1-432d-b3c7-0d4926b5fb85" (UID: "ba1e0e74-26d1-432d-b3c7-0d4926b5fb85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:07:31.828653 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:31.828634    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-home\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:07:31.828718 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:31.828659    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-tmp-dir\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:07:31.828718 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:31.828678    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba1e0e74-26d1-432d-b3c7-0d4926b5fb85" (UID: "ba1e0e74-26d1-432d-b3c7-0d4926b5fb85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:07:31.830431 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:31.830401    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba1e0e74-26d1-432d-b3c7-0d4926b5fb85" (UID: "ba1e0e74-26d1-432d-b3c7-0d4926b5fb85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:07:31.830431 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:31.830414    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-kube-api-access-lgk48" (OuterVolumeSpecName: "kube-api-access-lgk48") pod "ba1e0e74-26d1-432d-b3c7-0d4926b5fb85" (UID: "ba1e0e74-26d1-432d-b3c7-0d4926b5fb85"). InnerVolumeSpecName "kube-api-access-lgk48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:07:31.830570 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:31.830541    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-dshm" (OuterVolumeSpecName: "dshm") pod "ba1e0e74-26d1-432d-b3c7-0d4926b5fb85" (UID: "ba1e0e74-26d1-432d-b3c7-0d4926b5fb85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:07:31.929718 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:31.929682    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-tls-certs\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:07:31.929718 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:31.929712    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgk48\" (UniqueName: \"kubernetes.io/projected/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-kube-api-access-lgk48\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:07:31.929718 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:31.929722    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-dshm\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:07:31.929950 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:31.929732    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85-model-cache\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:07:32.651412 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:32.651378    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"
Sep 05 20:07:32.651870 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:32.651409    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq" event={"ID":"ba1e0e74-26d1-432d-b3c7-0d4926b5fb85","Type":"ContainerDied","Data":"14d2f4c9456427ccefe0e7de2e189dd2bb3e70087f0aae0a25e4b4803e965b38"}
Sep 05 20:07:32.651870 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:32.651458    2583 scope.go:117] "RemoveContainer" containerID="8783124a3a0686e165e008b3222f90ed7df04aaf77bd9514a1707c8236777e55"
Sep 05 20:07:32.679296 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:32.679265    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"]
Sep 05 20:07:32.685677 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:32.685651    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0cc31a/auth-test-model-0cc31a-kserve-9c8ddb79-46lqq"]
Sep 05 20:07:33.435570 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:33.435537    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba1e0e74-26d1-432d-b3c7-0d4926b5fb85" path="/var/lib/kubelet/pods/ba1e0e74-26d1-432d-b3c7-0d4926b5fb85/volumes"
Sep 05 20:07:39.662975 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:39.662928    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:07:39.663478 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:39.662993    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:07:39.676059 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:39.676034    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:07:39.708778 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:07:39.708747    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:08:02.620099 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.620060    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"]
Sep 05 20:08:02.620600 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.620511    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a73d41-6dea-4e61-bb71-bc3263eb4027" containerName="main"
Sep 05 20:08:02.620600 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.620524    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a73d41-6dea-4e61-bb71-bc3263eb4027" containerName="main"
Sep 05 20:08:02.620600 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.620540    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba1e0e74-26d1-432d-b3c7-0d4926b5fb85" containerName="main"
Sep 05 20:08:02.620600 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.620545    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba1e0e74-26d1-432d-b3c7-0d4926b5fb85" containerName="main"
Sep 05 20:08:02.620753 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.620604    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="27a73d41-6dea-4e61-bb71-bc3263eb4027" containerName="main"
Sep 05 20:08:02.620753 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.620614    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba1e0e74-26d1-432d-b3c7-0d4926b5fb85" containerName="main"
Sep 05 20:08:02.623755 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.623738    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:08:02.626819 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.626798    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0cc31a\"/\"openshift-service-ca.crt\""
Sep 05 20:08:02.627369 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.627350    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0cc31a\"/\"kube-root-ca.crt\""
Sep 05 20:08:02.627631 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.627613    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0cc31a\"/\"sub-test-model-0cc31a-kserve-self-signed-certs\""
Sep 05 20:08:02.627700 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.627619    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0cc31a\"/\"default-dockercfg-g4lbb\""
Sep 05 20:08:02.634487 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.634462    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"]
Sep 05 20:08:02.727225 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.727187    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-home\") pod \"sub-test-model-0cc31a-kserve-88ff69545-7j2t5\" (UID: \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:08:02.727410 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.727261    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-dshm\") pod \"sub-test-model-0cc31a-kserve-88ff69545-7j2t5\" (UID: \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:08:02.727410 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.727295    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbhlj\" (UniqueName: \"kubernetes.io/projected/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-kube-api-access-tbhlj\") pod \"sub-test-model-0cc31a-kserve-88ff69545-7j2t5\" (UID: \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:08:02.727410 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.727348    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-model-cache\") pod \"sub-test-model-0cc31a-kserve-88ff69545-7j2t5\" (UID: \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:08:02.727410 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.727365    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-tls-certs\") pod \"sub-test-model-0cc31a-kserve-88ff69545-7j2t5\" (UID: \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:08:02.727410 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.727390    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-tmp-dir\") pod \"sub-test-model-0cc31a-kserve-88ff69545-7j2t5\" (UID: \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:08:02.828488 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.828444    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-model-cache\") pod \"sub-test-model-0cc31a-kserve-88ff69545-7j2t5\" (UID: \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:08:02.828488 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.828493    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-tls-certs\") pod \"sub-test-model-0cc31a-kserve-88ff69545-7j2t5\" (UID: \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:08:02.828764 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.828534    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-tmp-dir\") pod \"sub-test-model-0cc31a-kserve-88ff69545-7j2t5\" (UID: \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:08:02.828764 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.828670    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-home\") pod \"sub-test-model-0cc31a-kserve-88ff69545-7j2t5\" (UID: \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:08:02.828905 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.828784    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-dshm\") pod \"sub-test-model-0cc31a-kserve-88ff69545-7j2t5\" (UID: \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:08:02.828905 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.828810    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbhlj\" (UniqueName: \"kubernetes.io/projected/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-kube-api-access-tbhlj\") pod \"sub-test-model-0cc31a-kserve-88ff69545-7j2t5\" (UID: \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:08:02.829013 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.828930    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-model-cache\") pod \"sub-test-model-0cc31a-kserve-88ff69545-7j2t5\" (UID: \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:08:02.829299 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.829266    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-tmp-dir\") pod \"sub-test-model-0cc31a-kserve-88ff69545-7j2t5\" (UID: \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:08:02.829412 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.829299    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-home\") pod \"sub-test-model-0cc31a-kserve-88ff69545-7j2t5\" (UID: \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:08:02.831293 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.831270    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-dshm\") pod \"sub-test-model-0cc31a-kserve-88ff69545-7j2t5\" (UID: \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:08:02.831387 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.831370    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-tls-certs\") pod \"sub-test-model-0cc31a-kserve-88ff69545-7j2t5\" (UID: \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:08:02.837403 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.837374    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbhlj\" (UniqueName: \"kubernetes.io/projected/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-kube-api-access-tbhlj\") pod \"sub-test-model-0cc31a-kserve-88ff69545-7j2t5\" (UID: \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:08:02.934394 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:02.934289    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:08:03.216321 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:03.216290    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"]
Sep 05 20:08:03.217885 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:08:03.217857    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84bfc6ba_ea6c_4879_93ad_0fa8891b2301.slice/crio-63e99863711a8b22c7e8ec3a5774148ecfef55e9a139f21f52c0611a492a28ea WatchSource:0}: Error finding container 63e99863711a8b22c7e8ec3a5774148ecfef55e9a139f21f52c0611a492a28ea: Status 404 returned error can't find the container with id 63e99863711a8b22c7e8ec3a5774148ecfef55e9a139f21f52c0611a492a28ea
Sep 05 20:08:03.847127 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:03.847074    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5" event={"ID":"84bfc6ba-ea6c-4879-93ad-0fa8891b2301","Type":"ContainerStarted","Data":"680b9eb8376b76e6793aa199e42af267fa7677958c40bc15e871ef6d9333d059"}
Sep 05 20:08:03.847511 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:03.847139    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5" event={"ID":"84bfc6ba-ea6c-4879-93ad-0fa8891b2301","Type":"ContainerStarted","Data":"63e99863711a8b22c7e8ec3a5774148ecfef55e9a139f21f52c0611a492a28ea"}
Sep 05 20:08:03.873998 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:03.873940    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5" podStartSLOduration=1.873922672 podStartE2EDuration="1.873922672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:08:03.873798613 +0000 UTC m=+1482.903430522" watchObservedRunningTime="2026-09-05 20:08:03.873922672 +0000 UTC m=+1482.903554577"
Sep 05 20:08:12.935284 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:12.935248    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:08:12.935854 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:12.935298    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:08:12.948122 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:12.948079    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:08:13.921298 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:13.921260    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:08:21.557259 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:21.557228    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf9tj_f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b/ovn-acl-logging/0.log"
Sep 05 20:08:21.562817 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:21.562795    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf9tj_f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b/ovn-acl-logging/0.log"
Sep 05 20:08:45.905138 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:45.905038    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7845-3e01b7"]
Sep 05 20:08:45.908666 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:45.908648    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7845-3e01b7"
Sep 05 20:08:45.910932 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:45.910909    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cqkks\""
Sep 05 20:08:45.916000 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:45.915976    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7845-3e01b7"]
Sep 05 20:08:45.933784 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:45.933751    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb7tl\" (UniqueName: \"kubernetes.io/projected/a96bb3b7-9bbd-4e08-bcf4-bd05d4c205aa-kube-api-access-cb7tl\") pod \"mt-curl-7845-3e01b7\" (UID: \"a96bb3b7-9bbd-4e08-bcf4-bd05d4c205aa\") " pod="opendatahub/mt-curl-7845-3e01b7"
Sep 05 20:08:46.034307 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:46.034270    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb7tl\" (UniqueName: \"kubernetes.io/projected/a96bb3b7-9bbd-4e08-bcf4-bd05d4c205aa-kube-api-access-cb7tl\") pod \"mt-curl-7845-3e01b7\" (UID: \"a96bb3b7-9bbd-4e08-bcf4-bd05d4c205aa\") " pod="opendatahub/mt-curl-7845-3e01b7"
Sep 05 20:08:46.042292 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:46.042265    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb7tl\" (UniqueName: \"kubernetes.io/projected/a96bb3b7-9bbd-4e08-bcf4-bd05d4c205aa-kube-api-access-cb7tl\") pod \"mt-curl-7845-3e01b7\" (UID: \"a96bb3b7-9bbd-4e08-bcf4-bd05d4c205aa\") " pod="opendatahub/mt-curl-7845-3e01b7"
Sep 05 20:08:46.219981 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:46.219890    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7845-3e01b7"
Sep 05 20:08:46.499263 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:46.499232    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7845-3e01b7"]
Sep 05 20:08:46.500198 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:08:46.500164    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda96bb3b7_9bbd_4e08_bcf4_bd05d4c205aa.slice/crio-0b0d20339a2e459c9376c424e6e763e14121847d08bb64550c72d7e9c7c49511 WatchSource:0}: Error finding container 0b0d20339a2e459c9376c424e6e763e14121847d08bb64550c72d7e9c7c49511: Status 404 returned error can't find the container with id 0b0d20339a2e459c9376c424e6e763e14121847d08bb64550c72d7e9c7c49511
Sep 05 20:08:46.501474 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:46.501454    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 20:08:47.121600 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:47.121568    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7845-3e01b7" event={"ID":"a96bb3b7-9bbd-4e08-bcf4-bd05d4c205aa","Type":"ContainerStarted","Data":"0b0d20339a2e459c9376c424e6e763e14121847d08bb64550c72d7e9c7c49511"}
Sep 05 20:08:48.129668 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:48.129627    2583 generic.go:356] "Generic (PLEG): container finished" podID="a96bb3b7-9bbd-4e08-bcf4-bd05d4c205aa" containerID="e97cdc539f6180ee75a230baface8b2f74748aa2ebc2ab340b40604e91896bf9" exitCode=0
Sep 05 20:08:48.130071 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:48.129715    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7845-3e01b7" event={"ID":"a96bb3b7-9bbd-4e08-bcf4-bd05d4c205aa","Type":"ContainerDied","Data":"e97cdc539f6180ee75a230baface8b2f74748aa2ebc2ab340b40604e91896bf9"}
Sep 05 20:08:49.284778 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:49.284752    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7845-3e01b7"
Sep 05 20:08:49.342928 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:49.342895    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7845-3e01b7_a96bb3b7-9bbd-4e08-bcf4-bd05d4c205aa/mt-curl-7845-3e01b7/0.log"
Sep 05 20:08:49.363308 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:49.363276    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb7tl\" (UniqueName: \"kubernetes.io/projected/a96bb3b7-9bbd-4e08-bcf4-bd05d4c205aa-kube-api-access-cb7tl\") pod \"a96bb3b7-9bbd-4e08-bcf4-bd05d4c205aa\" (UID: \"a96bb3b7-9bbd-4e08-bcf4-bd05d4c205aa\") "
Sep 05 20:08:49.365774 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:49.365747    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96bb3b7-9bbd-4e08-bcf4-bd05d4c205aa-kube-api-access-cb7tl" (OuterVolumeSpecName: "kube-api-access-cb7tl") pod "a96bb3b7-9bbd-4e08-bcf4-bd05d4c205aa" (UID: "a96bb3b7-9bbd-4e08-bcf4-bd05d4c205aa"). InnerVolumeSpecName "kube-api-access-cb7tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:08:49.370068 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:49.370045    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7845-3e01b7"]
Sep 05 20:08:49.373886 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:49.373863    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7845-3e01b7"]
Sep 05 20:08:49.435611 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:49.435534    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a96bb3b7-9bbd-4e08-bcf4-bd05d4c205aa" path="/var/lib/kubelet/pods/a96bb3b7-9bbd-4e08-bcf4-bd05d4c205aa/volumes"
Sep 05 20:08:49.465051 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:49.465020    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb7tl\" (UniqueName: \"kubernetes.io/projected/a96bb3b7-9bbd-4e08-bcf4-bd05d4c205aa-kube-api-access-cb7tl\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:08:50.143307 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:50.143282    2583 scope.go:117] "RemoveContainer" containerID="e97cdc539f6180ee75a230baface8b2f74748aa2ebc2ab340b40604e91896bf9"
Sep 05 20:08:50.143307 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:50.143298    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7845-3e01b7"
Sep 05 20:08:51.050484 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.050438    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5"]
Sep 05 20:08:51.051141 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.051100    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a96bb3b7-9bbd-4e08-bcf4-bd05d4c205aa" containerName="mt-curl-7845-3e01b7"
Sep 05 20:08:51.051305 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.051288    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a96bb3b7-9bbd-4e08-bcf4-bd05d4c205aa" containerName="mt-curl-7845-3e01b7"
Sep 05 20:08:51.051571 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.051543    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a96bb3b7-9bbd-4e08-bcf4-bd05d4c205aa" containerName="mt-curl-7845-3e01b7"
Sep 05 20:08:51.056323 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.056302    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5"
Sep 05 20:08:51.058678 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.058660    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1aa465\""
Sep 05 20:08:51.064577 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.064552    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5"]
Sep 05 20:08:51.079713 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.079683    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8bc6\" (UniqueName: \"kubernetes.io/projected/5535c1b7-2a70-4435-87bc-294043b36b0b-kube-api-access-c8bc6\") pod \"payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5\" (UID: \"5535c1b7-2a70-4435-87bc-294043b36b0b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5"
Sep 05 20:08:51.079899 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.079726    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5535c1b7-2a70-4435-87bc-294043b36b0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5\" (UID: \"5535c1b7-2a70-4435-87bc-294043b36b0b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5"
Sep 05 20:08:51.092366 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.092331    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6"]
Sep 05 20:08:51.095932 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.095916    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6"
Sep 05 20:08:51.098067 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.098045    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1aa465-dockercfg-pmz74\""
Sep 05 20:08:51.107956 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.107931    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6"]
Sep 05 20:08:51.180741 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.180696    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5535c1b7-2a70-4435-87bc-294043b36b0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5\" (UID: \"5535c1b7-2a70-4435-87bc-294043b36b0b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5"
Sep 05 20:08:51.180932 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.180888    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krqxp\" (UniqueName: \"kubernetes.io/projected/d2f9c325-e79d-47e6-be19-97f4fb4c71ea-kube-api-access-krqxp\") pod \"payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6\" (UID: \"d2f9c325-e79d-47e6-be19-97f4fb4c71ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6"
Sep 05 20:08:51.180932 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.180923    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f9c325-e79d-47e6-be19-97f4fb4c71ea-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6\" (UID: \"d2f9c325-e79d-47e6-be19-97f4fb4c71ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6"
Sep 05 20:08:51.181047 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.180964    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8bc6\" (UniqueName: \"kubernetes.io/projected/5535c1b7-2a70-4435-87bc-294043b36b0b-kube-api-access-c8bc6\") pod \"payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5\" (UID: \"5535c1b7-2a70-4435-87bc-294043b36b0b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5"
Sep 05 20:08:51.181526 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.181502    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5535c1b7-2a70-4435-87bc-294043b36b0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5\" (UID: \"5535c1b7-2a70-4435-87bc-294043b36b0b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5"
Sep 05 20:08:51.192122 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.192085    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8bc6\" (UniqueName: \"kubernetes.io/projected/5535c1b7-2a70-4435-87bc-294043b36b0b-kube-api-access-c8bc6\") pod \"payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5\" (UID: \"5535c1b7-2a70-4435-87bc-294043b36b0b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5"
Sep 05 20:08:51.282353 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.282310    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krqxp\" (UniqueName: \"kubernetes.io/projected/d2f9c325-e79d-47e6-be19-97f4fb4c71ea-kube-api-access-krqxp\") pod \"payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6\" (UID: \"d2f9c325-e79d-47e6-be19-97f4fb4c71ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6"
Sep 05 20:08:51.282353 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.282355    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f9c325-e79d-47e6-be19-97f4fb4c71ea-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6\" (UID: \"d2f9c325-e79d-47e6-be19-97f4fb4c71ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6"
Sep 05 20:08:51.283071 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.283049    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f9c325-e79d-47e6-be19-97f4fb4c71ea-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6\" (UID: \"d2f9c325-e79d-47e6-be19-97f4fb4c71ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6"
Sep 05 20:08:51.290070 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.290045    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krqxp\" (UniqueName: \"kubernetes.io/projected/d2f9c325-e79d-47e6-be19-97f4fb4c71ea-kube-api-access-krqxp\") pod \"payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6\" (UID: \"d2f9c325-e79d-47e6-be19-97f4fb4c71ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6"
Sep 05 20:08:51.370998 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.370955    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5"
Sep 05 20:08:51.407645 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.407610    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6"
Sep 05 20:08:51.650122 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.650077    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5"]
Sep 05 20:08:51.651551 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:08:51.651525    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5535c1b7_2a70_4435_87bc_294043b36b0b.slice/crio-74342b5b611760ff5f07992318a671c72f9be3fde2b62dc0c71cf369edc50ca8 WatchSource:0}: Error finding container 74342b5b611760ff5f07992318a671c72f9be3fde2b62dc0c71cf369edc50ca8: Status 404 returned error can't find the container with id 74342b5b611760ff5f07992318a671c72f9be3fde2b62dc0c71cf369edc50ca8
Sep 05 20:08:51.698994 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:51.698972    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6"]
Sep 05 20:08:51.700326 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:08:51.700285    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f9c325_e79d_47e6_be19_97f4fb4c71ea.slice/crio-6f6e2c19d047e10f111bb783f6f646a3f509e6f3d08ecb44769b6ab31c0b5232 WatchSource:0}: Error finding container 6f6e2c19d047e10f111bb783f6f646a3f509e6f3d08ecb44769b6ab31c0b5232: Status 404 returned error can't find the container with id 6f6e2c19d047e10f111bb783f6f646a3f509e6f3d08ecb44769b6ab31c0b5232
Sep 05 20:08:52.159659 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:52.159621    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5" event={"ID":"5535c1b7-2a70-4435-87bc-294043b36b0b","Type":"ContainerStarted","Data":"a5c4ad657a37ec1d0a798c652e52468737802d1f0d4f1889d619bd63b6d64c3b"}
Sep 05 20:08:52.160196 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:52.159684    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5" event={"ID":"5535c1b7-2a70-4435-87bc-294043b36b0b","Type":"ContainerStarted","Data":"74342b5b611760ff5f07992318a671c72f9be3fde2b62dc0c71cf369edc50ca8"}
Sep 05 20:08:52.160196 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:52.159815    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5"
Sep 05 20:08:52.161191 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:52.161169    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6" event={"ID":"d2f9c325-e79d-47e6-be19-97f4fb4c71ea","Type":"ContainerStarted","Data":"d626bbb68c6ddff7c8cb87dde9ed40afdfb379be359c0035c8a3580f449c1e49"}
Sep 05 20:08:52.161263 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:52.161208    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6" event={"ID":"d2f9c325-e79d-47e6-be19-97f4fb4c71ea","Type":"ContainerStarted","Data":"6f6e2c19d047e10f111bb783f6f646a3f509e6f3d08ecb44769b6ab31c0b5232"}
Sep 05 20:08:52.161263 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:52.161227    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6"
Sep 05 20:08:52.176871 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:52.176815    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5" podStartSLOduration=1.176800481 podStartE2EDuration="1.176800481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:08:52.175583149 +0000 UTC m=+1531.205215059" watchObservedRunningTime="2026-09-05 20:08:52.176800481 +0000 UTC m=+1531.206432387"
Sep 05 20:08:52.192234 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:52.192178    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6" podStartSLOduration=1.192162102 podStartE2EDuration="1.192162102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:08:52.191914615 +0000 UTC m=+1531.221546520" watchObservedRunningTime="2026-09-05 20:08:52.192162102 +0000 UTC m=+1531.221794007"
Sep 05 20:08:54.446908 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:54.446866    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"]
Sep 05 20:08:55.416709 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:08:55.416673    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"]
Sep 05 20:09:03.170771 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:03.170736    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5"
Sep 05 20:09:03.171287 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:03.171094    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6"
Sep 05 20:09:09.467871 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:09.467804    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg" podUID="a8aa7ef6-d244-49d2-bfc9-478d8fece05e" containerName="main" containerID="cri-o://74e4ad3e8978af686047b4725ff14c118197cc071922cb1e70e5ed6b6e4a7791" gracePeriod=45
Sep 05 20:09:09.725189 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:09.725124    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:09:09.871617 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:09.871580    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-home\") pod \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\" (UID: \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\") "
Sep 05 20:09:09.871764 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:09.871626    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-tls-certs\") pod \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\" (UID: \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\") "
Sep 05 20:09:09.871764 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:09.871710    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-dshm\") pod \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\" (UID: \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\") "
Sep 05 20:09:09.871871 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:09.871780    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-model-cache\") pod \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\" (UID: \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\") "
Sep 05 20:09:09.871871 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:09.871813    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjpzc\" (UniqueName: \"kubernetes.io/projected/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-kube-api-access-tjpzc\") pod \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\" (UID: \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\") "
Sep 05 20:09:09.871871 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:09.871858    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-tmp-dir\") pod \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\" (UID: \"a8aa7ef6-d244-49d2-bfc9-478d8fece05e\") "
Sep 05 20:09:09.871871 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:09.871861    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-home" (OuterVolumeSpecName: "home") pod "a8aa7ef6-d244-49d2-bfc9-478d8fece05e" (UID: "a8aa7ef6-d244-49d2-bfc9-478d8fece05e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:09:09.872077 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:09.872049    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-model-cache" (OuterVolumeSpecName: "model-cache") pod "a8aa7ef6-d244-49d2-bfc9-478d8fece05e" (UID: "a8aa7ef6-d244-49d2-bfc9-478d8fece05e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:09:09.872238 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:09.872211    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a8aa7ef6-d244-49d2-bfc9-478d8fece05e" (UID: "a8aa7ef6-d244-49d2-bfc9-478d8fece05e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:09:09.872324 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:09.872306    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-home\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:09:09.872397 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:09.872330    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-model-cache\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:09:09.872397 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:09.872378    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-tmp-dir\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:09:09.873914 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:09.873884    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-kube-api-access-tjpzc" (OuterVolumeSpecName: "kube-api-access-tjpzc") pod "a8aa7ef6-d244-49d2-bfc9-478d8fece05e" (UID: "a8aa7ef6-d244-49d2-bfc9-478d8fece05e"). InnerVolumeSpecName "kube-api-access-tjpzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:09:09.874029 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:09.873956    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-dshm" (OuterVolumeSpecName: "dshm") pod "a8aa7ef6-d244-49d2-bfc9-478d8fece05e" (UID: "a8aa7ef6-d244-49d2-bfc9-478d8fece05e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:09:09.874157 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:09.874129    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a8aa7ef6-d244-49d2-bfc9-478d8fece05e" (UID: "a8aa7ef6-d244-49d2-bfc9-478d8fece05e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:09:09.972884 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:09.972850    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-tls-certs\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:09:09.972884 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:09.972879    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-dshm\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:09:09.972884 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:09.972887    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjpzc\" (UniqueName: \"kubernetes.io/projected/a8aa7ef6-d244-49d2-bfc9-478d8fece05e-kube-api-access-tjpzc\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:09:10.280706 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.280665    2583 generic.go:356] "Generic (PLEG): container finished" podID="a8aa7ef6-d244-49d2-bfc9-478d8fece05e" containerID="74e4ad3e8978af686047b4725ff14c118197cc071922cb1e70e5ed6b6e4a7791" exitCode=0
Sep 05 20:09:10.280874 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.280710    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"
Sep 05 20:09:10.280874 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.280755    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg" event={"ID":"a8aa7ef6-d244-49d2-bfc9-478d8fece05e","Type":"ContainerDied","Data":"74e4ad3e8978af686047b4725ff14c118197cc071922cb1e70e5ed6b6e4a7791"}
Sep 05 20:09:10.280874 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.280794    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg" event={"ID":"a8aa7ef6-d244-49d2-bfc9-478d8fece05e","Type":"ContainerDied","Data":"be862ed7a9faae4b7b50c9ecdb783ece05c3e329cdcb4085e159aa7e3b739701"}
Sep 05 20:09:10.280874 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.280811    2583 scope.go:117] "RemoveContainer" containerID="74e4ad3e8978af686047b4725ff14c118197cc071922cb1e70e5ed6b6e4a7791"
Sep 05 20:09:10.290904 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.290885    2583 scope.go:117] "RemoveContainer" containerID="74e4ad3e8978af686047b4725ff14c118197cc071922cb1e70e5ed6b6e4a7791"
Sep 05 20:09:10.291225 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:09:10.291202    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e4ad3e8978af686047b4725ff14c118197cc071922cb1e70e5ed6b6e4a7791\": container with ID starting with 74e4ad3e8978af686047b4725ff14c118197cc071922cb1e70e5ed6b6e4a7791 not found: ID does not exist" containerID="74e4ad3e8978af686047b4725ff14c118197cc071922cb1e70e5ed6b6e4a7791"
Sep 05 20:09:10.291283 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.291235    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e4ad3e8978af686047b4725ff14c118197cc071922cb1e70e5ed6b6e4a7791"} err="failed to get container status \"74e4ad3e8978af686047b4725ff14c118197cc071922cb1e70e5ed6b6e4a7791\": rpc error: code = NotFound desc = could not find container \"74e4ad3e8978af686047b4725ff14c118197cc071922cb1e70e5ed6b6e4a7791\": container with ID starting with 74e4ad3e8978af686047b4725ff14c118197cc071922cb1e70e5ed6b6e4a7791 not found: ID does not exist"
Sep 05 20:09:10.303675 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.303646    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"]
Sep 05 20:09:10.309597 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.309575    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg"]
Sep 05 20:09:10.437898 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.437856    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5" podUID="84bfc6ba-ea6c-4879-93ad-0fa8891b2301" containerName="main" containerID="cri-o://680b9eb8376b76e6793aa199e42af267fa7677958c40bc15e871ef6d9333d059" gracePeriod=45
Sep 05 20:09:10.698320 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.698293    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:09:10.779707 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.779675    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-home\") pod \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\" (UID: \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\") "
Sep 05 20:09:10.779707 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.779710    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-tmp-dir\") pod \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\" (UID: \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\") "
Sep 05 20:09:10.779903 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.779766    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-dshm\") pod \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\" (UID: \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\") "
Sep 05 20:09:10.779903 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.779796    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-tls-certs\") pod \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\" (UID: \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\") "
Sep 05 20:09:10.779903 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.779837    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbhlj\" (UniqueName: \"kubernetes.io/projected/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-kube-api-access-tbhlj\") pod \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\" (UID: \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\") "
Sep 05 20:09:10.779903 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.779897    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-model-cache\") pod \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\" (UID: \"84bfc6ba-ea6c-4879-93ad-0fa8891b2301\") "
Sep 05 20:09:10.780080 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.779992    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-home" (OuterVolumeSpecName: "home") pod "84bfc6ba-ea6c-4879-93ad-0fa8891b2301" (UID: "84bfc6ba-ea6c-4879-93ad-0fa8891b2301"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:09:10.780166 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.780075    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "84bfc6ba-ea6c-4879-93ad-0fa8891b2301" (UID: "84bfc6ba-ea6c-4879-93ad-0fa8891b2301"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:09:10.780270 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.780249    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-model-cache" (OuterVolumeSpecName: "model-cache") pod "84bfc6ba-ea6c-4879-93ad-0fa8891b2301" (UID: "84bfc6ba-ea6c-4879-93ad-0fa8891b2301"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:09:10.780349 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.780331    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-model-cache\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:09:10.780401 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.780355    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-home\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:09:10.780401 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.780375    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-tmp-dir\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:09:10.781988 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.781961    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-dshm" (OuterVolumeSpecName: "dshm") pod "84bfc6ba-ea6c-4879-93ad-0fa8891b2301" (UID: "84bfc6ba-ea6c-4879-93ad-0fa8891b2301"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:09:10.782085 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.782029    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-kube-api-access-tbhlj" (OuterVolumeSpecName: "kube-api-access-tbhlj") pod "84bfc6ba-ea6c-4879-93ad-0fa8891b2301" (UID: "84bfc6ba-ea6c-4879-93ad-0fa8891b2301"). InnerVolumeSpecName "kube-api-access-tbhlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:09:10.782085 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.782077    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "84bfc6ba-ea6c-4879-93ad-0fa8891b2301" (UID: "84bfc6ba-ea6c-4879-93ad-0fa8891b2301"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:09:10.881013 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.880972    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-dshm\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:09:10.881013 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.881006    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-tls-certs\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:09:10.881013 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:10.881018    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbhlj\" (UniqueName: \"kubernetes.io/projected/84bfc6ba-ea6c-4879-93ad-0fa8891b2301-kube-api-access-tbhlj\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:09:11.289059 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:11.288977    2583 generic.go:356] "Generic (PLEG): container finished" podID="84bfc6ba-ea6c-4879-93ad-0fa8891b2301" containerID="680b9eb8376b76e6793aa199e42af267fa7677958c40bc15e871ef6d9333d059" exitCode=0
Sep 05 20:09:11.289059 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:11.289022    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"
Sep 05 20:09:11.289303 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:11.289058    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5" event={"ID":"84bfc6ba-ea6c-4879-93ad-0fa8891b2301","Type":"ContainerDied","Data":"680b9eb8376b76e6793aa199e42af267fa7677958c40bc15e871ef6d9333d059"}
Sep 05 20:09:11.289303 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:11.289099    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5" event={"ID":"84bfc6ba-ea6c-4879-93ad-0fa8891b2301","Type":"ContainerDied","Data":"63e99863711a8b22c7e8ec3a5774148ecfef55e9a139f21f52c0611a492a28ea"}
Sep 05 20:09:11.289303 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:11.289138    2583 scope.go:117] "RemoveContainer" containerID="680b9eb8376b76e6793aa199e42af267fa7677958c40bc15e871ef6d9333d059"
Sep 05 20:09:11.299088 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:11.299072    2583 scope.go:117] "RemoveContainer" containerID="680b9eb8376b76e6793aa199e42af267fa7677958c40bc15e871ef6d9333d059"
Sep 05 20:09:11.299374 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:09:11.299353    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"680b9eb8376b76e6793aa199e42af267fa7677958c40bc15e871ef6d9333d059\": container with ID starting with 680b9eb8376b76e6793aa199e42af267fa7677958c40bc15e871ef6d9333d059 not found: ID does not exist" containerID="680b9eb8376b76e6793aa199e42af267fa7677958c40bc15e871ef6d9333d059"
Sep 05 20:09:11.299423 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:11.299380    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"680b9eb8376b76e6793aa199e42af267fa7677958c40bc15e871ef6d9333d059"} err="failed to get container status \"680b9eb8376b76e6793aa199e42af267fa7677958c40bc15e871ef6d9333d059\": rpc error: code = NotFound desc = could not find container \"680b9eb8376b76e6793aa199e42af267fa7677958c40bc15e871ef6d9333d059\": container with ID starting with 680b9eb8376b76e6793aa199e42af267fa7677958c40bc15e871ef6d9333d059 not found: ID does not exist"
Sep 05 20:09:11.313639 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:11.313615    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"]
Sep 05 20:09:11.317601 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:11.317577    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0cc31a/sub-test-model-0cc31a-kserve-88ff69545-7j2t5"]
Sep 05 20:09:11.435179 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:11.435137    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84bfc6ba-ea6c-4879-93ad-0fa8891b2301" path="/var/lib/kubelet/pods/84bfc6ba-ea6c-4879-93ad-0fa8891b2301/volumes"
Sep 05 20:09:11.435639 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:11.435619    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8aa7ef6-d244-49d2-bfc9-478d8fece05e" path="/var/lib/kubelet/pods/a8aa7ef6-d244-49d2-bfc9-478d8fece05e/volumes"
Sep 05 20:09:14.698340 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:14.698284    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-4d1885/sub-test-model-4d1885-kserve-56969f4d8c-l5ssg" podUID="a8aa7ef6-d244-49d2-bfc9-478d8fece05e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 05 20:09:14.973552 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:14.973454    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"]
Sep 05 20:09:14.974095 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:14.974075    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8aa7ef6-d244-49d2-bfc9-478d8fece05e" containerName="main"
Sep 05 20:09:14.974095 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:14.974096    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8aa7ef6-d244-49d2-bfc9-478d8fece05e" containerName="main"
Sep 05 20:09:14.974316 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:14.974162    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84bfc6ba-ea6c-4879-93ad-0fa8891b2301" containerName="main"
Sep 05 20:09:14.974316 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:14.974171    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="84bfc6ba-ea6c-4879-93ad-0fa8891b2301" containerName="main"
Sep 05 20:09:14.974316 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:14.974293    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="84bfc6ba-ea6c-4879-93ad-0fa8891b2301" containerName="main"
Sep 05 20:09:14.974316 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:14.974308    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8aa7ef6-d244-49d2-bfc9-478d8fece05e" containerName="main"
Sep 05 20:09:14.979045 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:14.979022    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:14.981212 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:14.981146    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-dbbedd-openshift-default-dockercfg-n86rw\""
Sep 05 20:09:14.991314 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:14.991288    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"]
Sep 05 20:09:15.126021 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.125988    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4gmm\" (UniqueName: \"kubernetes.io/projected/9626662f-1234-4008-be31-7627e910a08d-kube-api-access-n4gmm\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.126225 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.126030    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-istio-data\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.126225 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.126060    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-workload-certs\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.126225 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.126097    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9626662f-1234-4008-be31-7627e910a08d-istio-podinfo\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.126225 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.126165    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9626662f-1234-4008-be31-7627e910a08d-istiod-ca-cert\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.126225 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.126218    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-credential-socket\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.126398 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.126240    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-istio-envoy\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.126398 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.126257    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-workload-socket\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.126398 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.126284    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9626662f-1234-4008-be31-7627e910a08d-istio-token\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.227411 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.227306    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4gmm\" (UniqueName: \"kubernetes.io/projected/9626662f-1234-4008-be31-7627e910a08d-kube-api-access-n4gmm\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.227411 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.227360    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-istio-data\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.227411 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.227381    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-workload-certs\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.227411 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.227408    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9626662f-1234-4008-be31-7627e910a08d-istio-podinfo\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.227742 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.227448    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9626662f-1234-4008-be31-7627e910a08d-istiod-ca-cert\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.227742 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.227494    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-credential-socket\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.227742 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.227526    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-istio-envoy\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.227742 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.227551    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-workload-socket\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.227742 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.227582    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9626662f-1234-4008-be31-7627e910a08d-istio-token\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.227996 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.227813    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-istio-data\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.227996 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.227862    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-workload-certs\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.227996 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.227954    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-credential-socket\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.227996 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.227981    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-workload-socket\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.228435 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.228413    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9626662f-1234-4008-be31-7627e910a08d-istiod-ca-cert\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.229819 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.229796    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-istio-envoy\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.229905 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.229890    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9626662f-1234-4008-be31-7627e910a08d-istio-podinfo\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.235532 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.235509    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9626662f-1234-4008-be31-7627e910a08d-istio-token\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.235716 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.235641    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4gmm\" (UniqueName: \"kubernetes.io/projected/9626662f-1234-4008-be31-7627e910a08d-kube-api-access-n4gmm\") pod \"e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") " pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.290914 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.290879    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:15.574836 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.574790    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"]
Sep 05 20:09:15.576746 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:09:15.576720    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9626662f_1234_4008_be31_7627e910a08d.slice/crio-e482f34a788a3b010ae7224a68a4f958285509c800ac4b58646d4ca6d28811b0 WatchSource:0}: Error finding container e482f34a788a3b010ae7224a68a4f958285509c800ac4b58646d4ca6d28811b0: Status 404 returned error can't find the container with id e482f34a788a3b010ae7224a68a4f958285509c800ac4b58646d4ca6d28811b0
Sep 05 20:09:15.578978 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.578946    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 20:09:15.579079 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.579010    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 20:09:15.579079 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:15.579040    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 20:09:16.323881 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:16.323844    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5" event={"ID":"9626662f-1234-4008-be31-7627e910a08d","Type":"ContainerStarted","Data":"e3511647a03be28d9b0d0ecf4001c00275988d583daf209ce849c90f7f55ec83"}
Sep 05 20:09:16.324332 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:16.323902    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5" event={"ID":"9626662f-1234-4008-be31-7627e910a08d","Type":"ContainerStarted","Data":"e482f34a788a3b010ae7224a68a4f958285509c800ac4b58646d4ca6d28811b0"}
Sep 05 20:09:16.346096 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:16.346043    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5" podStartSLOduration=2.346025962 podStartE2EDuration="2.346025962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:09:16.343467461 +0000 UTC m=+1555.373099369" watchObservedRunningTime="2026-09-05 20:09:16.346025962 +0000 UTC m=+1555.375657867"
Sep 05 20:09:17.291538 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:17.291505    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:17.296896 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:17.296867    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:17.329753 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:17.329724    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:17.330611 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:17.330592    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:20.133982 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:20.133945    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf"]
Sep 05 20:09:20.137996 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:20.137977    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf"
Sep 05 20:09:20.140341 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:20.140317    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-dbbedd\""
Sep 05 20:09:20.140341 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:20.140333    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-dbbedd-dockercfg-vxrqf\""
Sep 05 20:09:20.150627 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:20.150598    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf"]
Sep 05 20:09:20.276510 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:20.276475    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f30684-3a08-40db-b409-c76c14085fa9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf\" (UID: \"71f30684-3a08-40db-b409-c76c14085fa9\") " pod="openshift-ingress/payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf"
Sep 05 20:09:20.276694 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:20.276637    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q75qw\" (UniqueName: \"kubernetes.io/projected/71f30684-3a08-40db-b409-c76c14085fa9-kube-api-access-q75qw\") pod \"payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf\" (UID: \"71f30684-3a08-40db-b409-c76c14085fa9\") " pod="openshift-ingress/payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf"
Sep 05 20:09:20.377393 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:20.377356    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q75qw\" (UniqueName: \"kubernetes.io/projected/71f30684-3a08-40db-b409-c76c14085fa9-kube-api-access-q75qw\") pod \"payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf\" (UID: \"71f30684-3a08-40db-b409-c76c14085fa9\") " pod="openshift-ingress/payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf"
Sep 05 20:09:20.377575 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:20.377408    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f30684-3a08-40db-b409-c76c14085fa9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf\" (UID: \"71f30684-3a08-40db-b409-c76c14085fa9\") " pod="openshift-ingress/payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf"
Sep 05 20:09:20.378099 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:20.378079    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f30684-3a08-40db-b409-c76c14085fa9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf\" (UID: \"71f30684-3a08-40db-b409-c76c14085fa9\") " pod="openshift-ingress/payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf"
Sep 05 20:09:20.387282 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:20.387217    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q75qw\" (UniqueName: \"kubernetes.io/projected/71f30684-3a08-40db-b409-c76c14085fa9-kube-api-access-q75qw\") pod \"payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf\" (UID: \"71f30684-3a08-40db-b409-c76c14085fa9\") " pod="openshift-ingress/payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf"
Sep 05 20:09:20.452319 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:20.452282    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf"
Sep 05 20:09:20.735069 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:20.735027    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf"]
Sep 05 20:09:20.736808 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:09:20.736780    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f30684_3a08_40db_b409_c76c14085fa9.slice/crio-e5cf4dbfe28288a148c18f1158b7e45cb0697c622f17ea4e6b1afefe1519954a WatchSource:0}: Error finding container e5cf4dbfe28288a148c18f1158b7e45cb0697c622f17ea4e6b1afefe1519954a: Status 404 returned error can't find the container with id e5cf4dbfe28288a148c18f1158b7e45cb0697c622f17ea4e6b1afefe1519954a
Sep 05 20:09:21.358142 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:21.358079    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf" event={"ID":"71f30684-3a08-40db-b409-c76c14085fa9","Type":"ContainerStarted","Data":"b3def3b3258d5d3b3139778d99b11fc801fd4b2db07efc6fc17b8e55787305f0"}
Sep 05 20:09:21.358615 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:21.358169    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf"
Sep 05 20:09:21.358615 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:21.358192    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf" event={"ID":"71f30684-3a08-40db-b409-c76c14085fa9","Type":"ContainerStarted","Data":"e5cf4dbfe28288a148c18f1158b7e45cb0697c622f17ea4e6b1afefe1519954a"}
Sep 05 20:09:21.378608 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:21.378547    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf" podStartSLOduration=1.3785265789999999 podStartE2EDuration="1.378526579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:09:21.374781251 +0000 UTC m=+1560.404413156" watchObservedRunningTime="2026-09-05 20:09:21.378526579 +0000 UTC m=+1560.408158488"
Sep 05 20:09:21.806459 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:21.806374    2583 scope.go:117] "RemoveContainer" containerID="4dd2d18f3867a4527adad353ea75e0745056142f062481849bbb0f55890f5e84"
Sep 05 20:09:32.367200 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:32.367160    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf"
Sep 05 20:09:40.483964 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:40.483876    2583 generic.go:356] "Generic (PLEG): container finished" podID="9626662f-1234-4008-be31-7627e910a08d" containerID="e3511647a03be28d9b0d0ecf4001c00275988d583daf209ce849c90f7f55ec83" exitCode=0
Sep 05 20:09:40.483964 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:40.483928    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5" event={"ID":"9626662f-1234-4008-be31-7627e910a08d","Type":"ContainerDied","Data":"e3511647a03be28d9b0d0ecf4001c00275988d583daf209ce849c90f7f55ec83"}
Sep 05 20:09:40.484432 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:40.484354    2583 scope.go:117] "RemoveContainer" containerID="e3511647a03be28d9b0d0ecf4001c00275988d583daf209ce849c90f7f55ec83"
Sep 05 20:09:40.485547 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:40.485507    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 20:09:40.485641 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:40.485584    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 20:09:40.485641 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:40.485618    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 05 20:09:41.495873 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:41.495839    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5" event={"ID":"9626662f-1234-4008-be31-7627e910a08d","Type":"ContainerStarted","Data":"3e1a410ab6a5663446ff21c4354676b73d8f454e3abb7df7746b3650f2bb3b6f"}
Sep 05 20:09:42.291594 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:42.291542    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:42.292004 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:42.291971    2583 patch_prober.go:28] interesting pod/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Sep 05 20:09:42.292144 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:42.292055    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5" podUID="9626662f-1234-4008-be31-7627e910a08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Sep 05 20:09:43.291762 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:43.291728    2583 patch_prober.go:28] interesting pod/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Sep 05 20:09:43.292276 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:43.292234    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5" podUID="9626662f-1234-4008-be31-7627e910a08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Sep 05 20:09:44.291826 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:44.291698    2583 patch_prober.go:28] interesting pod/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Sep 05 20:09:44.292477 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:44.292447    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5" podUID="9626662f-1234-4008-be31-7627e910a08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Sep 05 20:09:45.291798 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:45.291756    2583 patch_prober.go:28] interesting pod/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Sep 05 20:09:45.292186 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:45.292151    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5" podUID="9626662f-1234-4008-be31-7627e910a08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Sep 05 20:09:46.001671 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:46.001634    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1aa465-9kd6r"]
Sep 05 20:09:46.004253 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:46.004235    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1aa465-9kd6r"
Sep 05 20:09:46.006724 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:46.006700    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 20:09:46.006808 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:46.006746    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zx89\""
Sep 05 20:09:46.013561 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:46.013539    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1aa465-9kd6r"]
Sep 05 20:09:46.140716 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:46.140669    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54619edc-8468-4820-9f36-4b0551298988-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1aa465-9kd6r\" (UID: \"54619edc-8468-4820-9f36-4b0551298988\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1aa465-9kd6r"
Sep 05 20:09:46.241491 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:46.241451    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54619edc-8468-4820-9f36-4b0551298988-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1aa465-9kd6r\" (UID: \"54619edc-8468-4820-9f36-4b0551298988\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1aa465-9kd6r"
Sep 05 20:09:46.242087 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:46.242068    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54619edc-8468-4820-9f36-4b0551298988-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1aa465-9kd6r\" (UID: \"54619edc-8468-4820-9f36-4b0551298988\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1aa465-9kd6r"
Sep 05 20:09:46.292291 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:46.292189    2583 patch_prober.go:28] interesting pod/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Sep 05 20:09:46.292291 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:46.292255    2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5" podUID="9626662f-1234-4008-be31-7627e910a08d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Sep 05 20:09:46.317037 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:46.317006    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1aa465-9kd6r"
Sep 05 20:09:46.599764 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:46.599720    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1aa465-9kd6r"]
Sep 05 20:09:47.295961 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:47.295871    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:47.296371 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:47.296150    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:47.296987 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:47.296966    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:09:47.533930 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:47.533892    2583 generic.go:356] "Generic (PLEG): container finished" podID="54619edc-8468-4820-9f36-4b0551298988" containerID="3a654a8d67a0c36752164ea88dca3a096f80bbc36e5aab3d5f7f70cb0f0a6e46" exitCode=0
Sep 05 20:09:47.534152 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:47.533936    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1aa465-9kd6r" event={"ID":"54619edc-8468-4820-9f36-4b0551298988","Type":"ContainerDied","Data":"3a654a8d67a0c36752164ea88dca3a096f80bbc36e5aab3d5f7f70cb0f0a6e46"}
Sep 05 20:09:47.534152 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:47.533987    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1aa465-9kd6r" event={"ID":"54619edc-8468-4820-9f36-4b0551298988","Type":"ContainerStarted","Data":"32b8c8b27764983723678badc965fb3b468db9aefbc4036329aef0c5bff22ffa"}
Sep 05 20:09:48.682667 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:48.682640    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1aa465-9kd6r"
Sep 05 20:09:48.770160 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:48.770125    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54619edc-8468-4820-9f36-4b0551298988-maas-api-service-ca\") pod \"54619edc-8468-4820-9f36-4b0551298988\" (UID: \"54619edc-8468-4820-9f36-4b0551298988\") "
Sep 05 20:09:48.770569 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:48.770541    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54619edc-8468-4820-9f36-4b0551298988-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "54619edc-8468-4820-9f36-4b0551298988" (UID: "54619edc-8468-4820-9f36-4b0551298988"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:09:48.871247 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:48.871197    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54619edc-8468-4820-9f36-4b0551298988-maas-api-service-ca\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:09:49.548431 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:49.548393    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1aa465-9kd6r"
Sep 05 20:09:49.548431 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:49.548423    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1aa465-9kd6r" event={"ID":"54619edc-8468-4820-9f36-4b0551298988","Type":"ContainerDied","Data":"32b8c8b27764983723678badc965fb3b468db9aefbc4036329aef0c5bff22ffa"}
Sep 05 20:09:49.548634 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:49.548471    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32b8c8b27764983723678badc965fb3b468db9aefbc4036329aef0c5bff22ffa"
Sep 05 20:09:53.126825 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.126766    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"]
Sep 05 20:09:53.128135 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.128065    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54619edc-8468-4820-9f36-4b0551298988" containerName="revoke-keys"
Sep 05 20:09:53.128135 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.128095    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="54619edc-8468-4820-9f36-4b0551298988" containerName="revoke-keys"
Sep 05 20:09:53.128334 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.128240    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="54619edc-8468-4820-9f36-4b0551298988" containerName="revoke-keys"
Sep 05 20:09:53.131495 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.131476    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:09:53.134894 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.134872    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0cc31a\"/\"default-dockercfg-g4lbb\""
Sep 05 20:09:53.135008 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.134878    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0cc31a\"/\"rate-test-model-0cc31a-kserve-self-signed-certs\""
Sep 05 20:09:53.135226 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.135207    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0cc31a\"/\"openshift-service-ca.crt\""
Sep 05 20:09:53.135356 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.135322    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0cc31a\"/\"kube-root-ca.crt\""
Sep 05 20:09:53.141792 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.141771    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"]
Sep 05 20:09:53.318779 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.318743    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bec1fad-754c-48c4-bba6-b9fc839a1709-dshm\") pod \"rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd\" (UID: \"1bec1fad-754c-48c4-bba6-b9fc839a1709\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:09:53.318779 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.318784    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bec1fad-754c-48c4-bba6-b9fc839a1709-tmp-dir\") pod \"rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd\" (UID: \"1bec1fad-754c-48c4-bba6-b9fc839a1709\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:09:53.318988 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.318814    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9dxv\" (UniqueName: \"kubernetes.io/projected/1bec1fad-754c-48c4-bba6-b9fc839a1709-kube-api-access-r9dxv\") pod \"rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd\" (UID: \"1bec1fad-754c-48c4-bba6-b9fc839a1709\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:09:53.318988 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.318922    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bec1fad-754c-48c4-bba6-b9fc839a1709-tls-certs\") pod \"rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd\" (UID: \"1bec1fad-754c-48c4-bba6-b9fc839a1709\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:09:53.318988 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.318969    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bec1fad-754c-48c4-bba6-b9fc839a1709-home\") pod \"rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd\" (UID: \"1bec1fad-754c-48c4-bba6-b9fc839a1709\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:09:53.319167 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.319041    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bec1fad-754c-48c4-bba6-b9fc839a1709-model-cache\") pod \"rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd\" (UID: \"1bec1fad-754c-48c4-bba6-b9fc839a1709\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:09:53.419668 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.419563    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bec1fad-754c-48c4-bba6-b9fc839a1709-home\") pod \"rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd\" (UID: \"1bec1fad-754c-48c4-bba6-b9fc839a1709\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:09:53.419668 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.419615    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bec1fad-754c-48c4-bba6-b9fc839a1709-model-cache\") pod \"rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd\" (UID: \"1bec1fad-754c-48c4-bba6-b9fc839a1709\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:09:53.419668 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.419653    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bec1fad-754c-48c4-bba6-b9fc839a1709-dshm\") pod \"rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd\" (UID: \"1bec1fad-754c-48c4-bba6-b9fc839a1709\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:09:53.419968 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.419689    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bec1fad-754c-48c4-bba6-b9fc839a1709-tmp-dir\") pod \"rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd\" (UID: \"1bec1fad-754c-48c4-bba6-b9fc839a1709\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:09:53.419968 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.419730    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9dxv\" (UniqueName: \"kubernetes.io/projected/1bec1fad-754c-48c4-bba6-b9fc839a1709-kube-api-access-r9dxv\") pod \"rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd\" (UID: \"1bec1fad-754c-48c4-bba6-b9fc839a1709\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:09:53.419968 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.419815    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bec1fad-754c-48c4-bba6-b9fc839a1709-tls-certs\") pod \"rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd\" (UID: \"1bec1fad-754c-48c4-bba6-b9fc839a1709\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:09:53.420104 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.420033    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bec1fad-754c-48c4-bba6-b9fc839a1709-home\") pod \"rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd\" (UID: \"1bec1fad-754c-48c4-bba6-b9fc839a1709\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:09:53.420104 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.420061    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bec1fad-754c-48c4-bba6-b9fc839a1709-tmp-dir\") pod \"rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd\" (UID: \"1bec1fad-754c-48c4-bba6-b9fc839a1709\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:09:53.420104 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.420084    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bec1fad-754c-48c4-bba6-b9fc839a1709-model-cache\") pod \"rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd\" (UID: \"1bec1fad-754c-48c4-bba6-b9fc839a1709\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:09:53.422103 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.422077    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bec1fad-754c-48c4-bba6-b9fc839a1709-dshm\") pod \"rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd\" (UID: \"1bec1fad-754c-48c4-bba6-b9fc839a1709\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:09:53.422318 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.422298    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bec1fad-754c-48c4-bba6-b9fc839a1709-tls-certs\") pod \"rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd\" (UID: \"1bec1fad-754c-48c4-bba6-b9fc839a1709\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:09:53.428338 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.428315    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9dxv\" (UniqueName: \"kubernetes.io/projected/1bec1fad-754c-48c4-bba6-b9fc839a1709-kube-api-access-r9dxv\") pod \"rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd\" (UID: \"1bec1fad-754c-48c4-bba6-b9fc839a1709\") " pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:09:53.442086 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.442047    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:09:53.725795 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:53.725705    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"]
Sep 05 20:09:53.729086 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:09:53.729056    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bec1fad_754c_48c4_bba6_b9fc839a1709.slice/crio-4527e77f792e60a8b682055e0d50bbd0b0ffe57354ef2a3c465db28f2aea8013 WatchSource:0}: Error finding container 4527e77f792e60a8b682055e0d50bbd0b0ffe57354ef2a3c465db28f2aea8013: Status 404 returned error can't find the container with id 4527e77f792e60a8b682055e0d50bbd0b0ffe57354ef2a3c465db28f2aea8013
Sep 05 20:09:54.583470 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:54.583433    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd" event={"ID":"1bec1fad-754c-48c4-bba6-b9fc839a1709","Type":"ContainerStarted","Data":"66dc1e89e58dd50cf0f64845af1cdcd0085b1e1f7f465a19b12198f639b0262c"}
Sep 05 20:09:54.583862 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:54.583478    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd" event={"ID":"1bec1fad-754c-48c4-bba6-b9fc839a1709","Type":"ContainerStarted","Data":"4527e77f792e60a8b682055e0d50bbd0b0ffe57354ef2a3c465db28f2aea8013"}
Sep 05 20:09:54.601629 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:09:54.601576    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd" podStartSLOduration=1.601558105 podStartE2EDuration="1.601558105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:09:54.599982134 +0000 UTC m=+1593.629614041" watchObservedRunningTime="2026-09-05 20:09:54.601558105 +0000 UTC m=+1593.631190010"
Sep 05 20:10:01.524441 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:01.524399    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6"]
Sep 05 20:10:01.524804 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:01.524659    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6" podUID="d2f9c325-e79d-47e6-be19-97f4fb4c71ea" containerName="payload-processing" containerID="cri-o://d626bbb68c6ddff7c8cb87dde9ed40afdfb379be359c0035c8a3580f449c1e49" gracePeriod=30
Sep 05 20:10:01.532247 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:01.532214    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5"]
Sep 05 20:10:01.532487 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:01.532453    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5" podUID="5535c1b7-2a70-4435-87bc-294043b36b0b" containerName="payload-pre-processing" containerID="cri-o://a5c4ad657a37ec1d0a798c652e52468737802d1f0d4f1889d619bd63b6d64c3b" gracePeriod=30
Sep 05 20:10:01.683441 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:01.683384    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1aa465-9kd6r"]
Sep 05 20:10:01.687256 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:01.687225    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1aa465-9kd6r"]
Sep 05 20:10:01.818722 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:01.818694    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6"
Sep 05 20:10:01.824270 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:01.824245    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5"
Sep 05 20:10:01.900350 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:01.900315    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krqxp\" (UniqueName: \"kubernetes.io/projected/d2f9c325-e79d-47e6-be19-97f4fb4c71ea-kube-api-access-krqxp\") pod \"d2f9c325-e79d-47e6-be19-97f4fb4c71ea\" (UID: \"d2f9c325-e79d-47e6-be19-97f4fb4c71ea\") "
Sep 05 20:10:01.900529 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:01.900383    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5535c1b7-2a70-4435-87bc-294043b36b0b-plugins-config-volume\") pod \"5535c1b7-2a70-4435-87bc-294043b36b0b\" (UID: \"5535c1b7-2a70-4435-87bc-294043b36b0b\") "
Sep 05 20:10:01.900529 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:01.900436    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f9c325-e79d-47e6-be19-97f4fb4c71ea-plugins-config-volume\") pod \"d2f9c325-e79d-47e6-be19-97f4fb4c71ea\" (UID: \"d2f9c325-e79d-47e6-be19-97f4fb4c71ea\") "
Sep 05 20:10:01.900529 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:01.900461    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8bc6\" (UniqueName: \"kubernetes.io/projected/5535c1b7-2a70-4435-87bc-294043b36b0b-kube-api-access-c8bc6\") pod \"5535c1b7-2a70-4435-87bc-294043b36b0b\" (UID: \"5535c1b7-2a70-4435-87bc-294043b36b0b\") "
Sep 05 20:10:01.900797 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:01.900765    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5535c1b7-2a70-4435-87bc-294043b36b0b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5535c1b7-2a70-4435-87bc-294043b36b0b" (UID: "5535c1b7-2a70-4435-87bc-294043b36b0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:10:01.900903 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:01.900837    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f9c325-e79d-47e6-be19-97f4fb4c71ea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d2f9c325-e79d-47e6-be19-97f4fb4c71ea" (UID: "d2f9c325-e79d-47e6-be19-97f4fb4c71ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:10:01.902638 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:01.902602    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5535c1b7-2a70-4435-87bc-294043b36b0b-kube-api-access-c8bc6" (OuterVolumeSpecName: "kube-api-access-c8bc6") pod "5535c1b7-2a70-4435-87bc-294043b36b0b" (UID: "5535c1b7-2a70-4435-87bc-294043b36b0b"). InnerVolumeSpecName "kube-api-access-c8bc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:10:01.903042 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:01.903016    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f9c325-e79d-47e6-be19-97f4fb4c71ea-kube-api-access-krqxp" (OuterVolumeSpecName: "kube-api-access-krqxp") pod "d2f9c325-e79d-47e6-be19-97f4fb4c71ea" (UID: "d2f9c325-e79d-47e6-be19-97f4fb4c71ea"). InnerVolumeSpecName "kube-api-access-krqxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:10:02.001592 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:02.001553    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2f9c325-e79d-47e6-be19-97f4fb4c71ea-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:02.001592 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:02.001586    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8bc6\" (UniqueName: \"kubernetes.io/projected/5535c1b7-2a70-4435-87bc-294043b36b0b-kube-api-access-c8bc6\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:02.001592 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:02.001595    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krqxp\" (UniqueName: \"kubernetes.io/projected/d2f9c325-e79d-47e6-be19-97f4fb4c71ea-kube-api-access-krqxp\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:02.001824 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:02.001607    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5535c1b7-2a70-4435-87bc-294043b36b0b-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:02.644948 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:02.644907    2583 generic.go:356] "Generic (PLEG): container finished" podID="5535c1b7-2a70-4435-87bc-294043b36b0b" containerID="a5c4ad657a37ec1d0a798c652e52468737802d1f0d4f1889d619bd63b6d64c3b" exitCode=0
Sep 05 20:10:02.645424 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:02.644967    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5"
Sep 05 20:10:02.645424 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:02.644992    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5" event={"ID":"5535c1b7-2a70-4435-87bc-294043b36b0b","Type":"ContainerDied","Data":"a5c4ad657a37ec1d0a798c652e52468737802d1f0d4f1889d619bd63b6d64c3b"}
Sep 05 20:10:02.645424 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:02.645091    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5" event={"ID":"5535c1b7-2a70-4435-87bc-294043b36b0b","Type":"ContainerDied","Data":"74342b5b611760ff5f07992318a671c72f9be3fde2b62dc0c71cf369edc50ca8"}
Sep 05 20:10:02.645424 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:02.645139    2583 scope.go:117] "RemoveContainer" containerID="a5c4ad657a37ec1d0a798c652e52468737802d1f0d4f1889d619bd63b6d64c3b"
Sep 05 20:10:02.646567 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:02.646526    2583 generic.go:356] "Generic (PLEG): container finished" podID="d2f9c325-e79d-47e6-be19-97f4fb4c71ea" containerID="d626bbb68c6ddff7c8cb87dde9ed40afdfb379be359c0035c8a3580f449c1e49" exitCode=0
Sep 05 20:10:02.646645 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:02.646583    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6"
Sep 05 20:10:02.646645 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:02.646617    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6" event={"ID":"d2f9c325-e79d-47e6-be19-97f4fb4c71ea","Type":"ContainerDied","Data":"d626bbb68c6ddff7c8cb87dde9ed40afdfb379be359c0035c8a3580f449c1e49"}
Sep 05 20:10:02.646745 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:02.646676    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6" event={"ID":"d2f9c325-e79d-47e6-be19-97f4fb4c71ea","Type":"ContainerDied","Data":"6f6e2c19d047e10f111bb783f6f646a3f509e6f3d08ecb44769b6ab31c0b5232"}
Sep 05 20:10:02.656005 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:02.655979    2583 scope.go:117] "RemoveContainer" containerID="a5c4ad657a37ec1d0a798c652e52468737802d1f0d4f1889d619bd63b6d64c3b"
Sep 05 20:10:02.656403 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:10:02.656383    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5c4ad657a37ec1d0a798c652e52468737802d1f0d4f1889d619bd63b6d64c3b\": container with ID starting with a5c4ad657a37ec1d0a798c652e52468737802d1f0d4f1889d619bd63b6d64c3b not found: ID does not exist" containerID="a5c4ad657a37ec1d0a798c652e52468737802d1f0d4f1889d619bd63b6d64c3b"
Sep 05 20:10:02.656492 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:02.656411    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5c4ad657a37ec1d0a798c652e52468737802d1f0d4f1889d619bd63b6d64c3b"} err="failed to get container status \"a5c4ad657a37ec1d0a798c652e52468737802d1f0d4f1889d619bd63b6d64c3b\": rpc error: code = NotFound desc = could not find container \"a5c4ad657a37ec1d0a798c652e52468737802d1f0d4f1889d619bd63b6d64c3b\": container with ID starting with a5c4ad657a37ec1d0a798c652e52468737802d1f0d4f1889d619bd63b6d64c3b not found: ID does not exist"
Sep 05 20:10:02.656492 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:02.656437    2583 scope.go:117] "RemoveContainer" containerID="d626bbb68c6ddff7c8cb87dde9ed40afdfb379be359c0035c8a3580f449c1e49"
Sep 05 20:10:02.666223 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:02.666200    2583 scope.go:117] "RemoveContainer" containerID="d626bbb68c6ddff7c8cb87dde9ed40afdfb379be359c0035c8a3580f449c1e49"
Sep 05 20:10:02.666603 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:10:02.666578    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d626bbb68c6ddff7c8cb87dde9ed40afdfb379be359c0035c8a3580f449c1e49\": container with ID starting with d626bbb68c6ddff7c8cb87dde9ed40afdfb379be359c0035c8a3580f449c1e49 not found: ID does not exist" containerID="d626bbb68c6ddff7c8cb87dde9ed40afdfb379be359c0035c8a3580f449c1e49"
Sep 05 20:10:02.666814 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:02.666608    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d626bbb68c6ddff7c8cb87dde9ed40afdfb379be359c0035c8a3580f449c1e49"} err="failed to get container status \"d626bbb68c6ddff7c8cb87dde9ed40afdfb379be359c0035c8a3580f449c1e49\": rpc error: code = NotFound desc = could not find container \"d626bbb68c6ddff7c8cb87dde9ed40afdfb379be359c0035c8a3580f449c1e49\": container with ID starting with d626bbb68c6ddff7c8cb87dde9ed40afdfb379be359c0035c8a3580f449c1e49 not found: ID does not exist"
Sep 05 20:10:02.677892 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:02.677851    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5"]
Sep 05 20:10:02.685005 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:02.684969    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1aa465-664646c578-v7jw5"]
Sep 05 20:10:02.695728 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:02.695693    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6"]
Sep 05 20:10:02.699875 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:02.699844    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1aa465-5b7cdd794c-xn2d6"]
Sep 05 20:10:03.435994 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:03.435945    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54619edc-8468-4820-9f36-4b0551298988" path="/var/lib/kubelet/pods/54619edc-8468-4820-9f36-4b0551298988/volumes"
Sep 05 20:10:03.436456 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:03.436434    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5535c1b7-2a70-4435-87bc-294043b36b0b" path="/var/lib/kubelet/pods/5535c1b7-2a70-4435-87bc-294043b36b0b/volumes"
Sep 05 20:10:03.436836 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:03.436817    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2f9c325-e79d-47e6-be19-97f4fb4c71ea" path="/var/lib/kubelet/pods/d2f9c325-e79d-47e6-be19-97f4fb4c71ea/volumes"
Sep 05 20:10:03.442913 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:03.442887    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:10:03.443023 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:03.442933    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:10:03.455724 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:03.455693    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:10:03.665851 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:03.665821    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:10:08.493895 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:08.493862    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbbedd-7swzk"]
Sep 05 20:10:08.494373 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:08.494357    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5535c1b7-2a70-4435-87bc-294043b36b0b" containerName="payload-pre-processing"
Sep 05 20:10:08.494427 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:08.494375    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5535c1b7-2a70-4435-87bc-294043b36b0b" containerName="payload-pre-processing"
Sep 05 20:10:08.494427 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:08.494382    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f9c325-e79d-47e6-be19-97f4fb4c71ea" containerName="payload-processing"
Sep 05 20:10:08.494427 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:08.494388    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f9c325-e79d-47e6-be19-97f4fb4c71ea" containerName="payload-processing"
Sep 05 20:10:08.494538 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:08.494455    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5535c1b7-2a70-4435-87bc-294043b36b0b" containerName="payload-pre-processing"
Sep 05 20:10:08.494538 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:08.494465    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f9c325-e79d-47e6-be19-97f4fb4c71ea" containerName="payload-processing"
Sep 05 20:10:08.498995 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:08.498975    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbbedd-7swzk"
Sep 05 20:10:08.501518 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:08.501499    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 20:10:08.501866 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:08.501839    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zx89\""
Sep 05 20:10:08.504642 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:08.504620    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbbedd-7swzk"]
Sep 05 20:10:08.565286 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:08.565249    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6c77af-2ef1-4839-8093-e8c04478274f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dbbedd-7swzk\" (UID: \"3c6c77af-2ef1-4839-8093-e8c04478274f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbbedd-7swzk"
Sep 05 20:10:08.666654 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:08.666620    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6c77af-2ef1-4839-8093-e8c04478274f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dbbedd-7swzk\" (UID: \"3c6c77af-2ef1-4839-8093-e8c04478274f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbbedd-7swzk"
Sep 05 20:10:08.667372 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:08.667345    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6c77af-2ef1-4839-8093-e8c04478274f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dbbedd-7swzk\" (UID: \"3c6c77af-2ef1-4839-8093-e8c04478274f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbbedd-7swzk"
Sep 05 20:10:08.811468 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:08.811366    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbbedd-7swzk"
Sep 05 20:10:09.086858 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:09.086828    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbbedd-7swzk"]
Sep 05 20:10:09.087686 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:10:09.087657    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6c77af_2ef1_4839_8093_e8c04478274f.slice/crio-0a4b0e8ea222df70cbb50b0a21202cac06876a2fd78f2edd39909fc5b3a275ce WatchSource:0}: Error finding container 0a4b0e8ea222df70cbb50b0a21202cac06876a2fd78f2edd39909fc5b3a275ce: Status 404 returned error can't find the container with id 0a4b0e8ea222df70cbb50b0a21202cac06876a2fd78f2edd39909fc5b3a275ce
Sep 05 20:10:09.693943 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:09.693848    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbbedd-7swzk" event={"ID":"3c6c77af-2ef1-4839-8093-e8c04478274f","Type":"ContainerStarted","Data":"2acd4002a6fa805571ac7c95231d840f1eb3a7139decbf7315714265cba17b9e"}
Sep 05 20:10:09.693943 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:09.693903    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbbedd-7swzk" event={"ID":"3c6c77af-2ef1-4839-8093-e8c04478274f","Type":"ContainerStarted","Data":"0a4b0e8ea222df70cbb50b0a21202cac06876a2fd78f2edd39909fc5b3a275ce"}
Sep 05 20:10:09.710750 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:09.710690    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbbedd-7swzk" podStartSLOduration=1.710675049 podStartE2EDuration="1.710675049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:10:09.708668214 +0000 UTC m=+1608.738300119" watchObservedRunningTime="2026-09-05 20:10:09.710675049 +0000 UTC m=+1608.740306954"
Sep 05 20:10:10.702447 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:10.702407    2583 generic.go:356] "Generic (PLEG): container finished" podID="3c6c77af-2ef1-4839-8093-e8c04478274f" containerID="2acd4002a6fa805571ac7c95231d840f1eb3a7139decbf7315714265cba17b9e" exitCode=0
Sep 05 20:10:10.702825 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:10.702452    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbbedd-7swzk" event={"ID":"3c6c77af-2ef1-4839-8093-e8c04478274f","Type":"ContainerDied","Data":"2acd4002a6fa805571ac7c95231d840f1eb3a7139decbf7315714265cba17b9e"}
Sep 05 20:10:11.854410 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:11.854383    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbbedd-7swzk"
Sep 05 20:10:11.899677 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:11.899639    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6c77af-2ef1-4839-8093-e8c04478274f-maas-api-service-ca\") pod \"3c6c77af-2ef1-4839-8093-e8c04478274f\" (UID: \"3c6c77af-2ef1-4839-8093-e8c04478274f\") "
Sep 05 20:10:11.900020 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:11.899997    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c6c77af-2ef1-4839-8093-e8c04478274f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3c6c77af-2ef1-4839-8093-e8c04478274f" (UID: "3c6c77af-2ef1-4839-8093-e8c04478274f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:10:12.001482 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:12.001397    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6c77af-2ef1-4839-8093-e8c04478274f-maas-api-service-ca\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:12.716455 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:12.716410    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbbedd-7swzk" event={"ID":"3c6c77af-2ef1-4839-8093-e8c04478274f","Type":"ContainerDied","Data":"0a4b0e8ea222df70cbb50b0a21202cac06876a2fd78f2edd39909fc5b3a275ce"}
Sep 05 20:10:12.716626 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:12.716461    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbbedd-7swzk"
Sep 05 20:10:12.716626 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:12.716515    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a4b0e8ea222df70cbb50b0a21202cac06876a2fd78f2edd39909fc5b3a275ce"
Sep 05 20:10:24.330705 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:24.330670    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf"]
Sep 05 20:10:24.331238 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:24.330966    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf" podUID="71f30684-3a08-40db-b409-c76c14085fa9" containerName="payload-processing" containerID="cri-o://b3def3b3258d5d3b3139778d99b11fc801fd4b2db07efc6fc17b8e55787305f0" gracePeriod=30
Sep 05 20:10:24.491596 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:24.491552    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbbedd-7swzk"]
Sep 05 20:10:24.494253 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:24.494228    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbbedd-7swzk"]
Sep 05 20:10:24.601835 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:24.601806    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf"
Sep 05 20:10:24.731999 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:24.731961    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f30684-3a08-40db-b409-c76c14085fa9-plugins-config-volume\") pod \"71f30684-3a08-40db-b409-c76c14085fa9\" (UID: \"71f30684-3a08-40db-b409-c76c14085fa9\") "
Sep 05 20:10:24.732228 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:24.732077    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q75qw\" (UniqueName: \"kubernetes.io/projected/71f30684-3a08-40db-b409-c76c14085fa9-kube-api-access-q75qw\") pod \"71f30684-3a08-40db-b409-c76c14085fa9\" (UID: \"71f30684-3a08-40db-b409-c76c14085fa9\") "
Sep 05 20:10:24.732419 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:24.732390    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f30684-3a08-40db-b409-c76c14085fa9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "71f30684-3a08-40db-b409-c76c14085fa9" (UID: "71f30684-3a08-40db-b409-c76c14085fa9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:10:24.734193 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:24.734172    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f30684-3a08-40db-b409-c76c14085fa9-kube-api-access-q75qw" (OuterVolumeSpecName: "kube-api-access-q75qw") pod "71f30684-3a08-40db-b409-c76c14085fa9" (UID: "71f30684-3a08-40db-b409-c76c14085fa9"). InnerVolumeSpecName "kube-api-access-q75qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:10:24.791447 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:24.791409    2583 generic.go:356] "Generic (PLEG): container finished" podID="71f30684-3a08-40db-b409-c76c14085fa9" containerID="b3def3b3258d5d3b3139778d99b11fc801fd4b2db07efc6fc17b8e55787305f0" exitCode=0
Sep 05 20:10:24.791622 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:24.791453    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf"
Sep 05 20:10:24.791622 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:24.791501    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf" event={"ID":"71f30684-3a08-40db-b409-c76c14085fa9","Type":"ContainerDied","Data":"b3def3b3258d5d3b3139778d99b11fc801fd4b2db07efc6fc17b8e55787305f0"}
Sep 05 20:10:24.791622 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:24.791564    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf" event={"ID":"71f30684-3a08-40db-b409-c76c14085fa9","Type":"ContainerDied","Data":"e5cf4dbfe28288a148c18f1158b7e45cb0697c622f17ea4e6b1afefe1519954a"}
Sep 05 20:10:24.791622 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:24.791593    2583 scope.go:117] "RemoveContainer" containerID="b3def3b3258d5d3b3139778d99b11fc801fd4b2db07efc6fc17b8e55787305f0"
Sep 05 20:10:24.801312 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:24.801287    2583 scope.go:117] "RemoveContainer" containerID="b3def3b3258d5d3b3139778d99b11fc801fd4b2db07efc6fc17b8e55787305f0"
Sep 05 20:10:24.801642 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:10:24.801618    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3def3b3258d5d3b3139778d99b11fc801fd4b2db07efc6fc17b8e55787305f0\": container with ID starting with b3def3b3258d5d3b3139778d99b11fc801fd4b2db07efc6fc17b8e55787305f0 not found: ID does not exist" containerID="b3def3b3258d5d3b3139778d99b11fc801fd4b2db07efc6fc17b8e55787305f0"
Sep 05 20:10:24.801729 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:24.801650    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3def3b3258d5d3b3139778d99b11fc801fd4b2db07efc6fc17b8e55787305f0"} err="failed to get container status \"b3def3b3258d5d3b3139778d99b11fc801fd4b2db07efc6fc17b8e55787305f0\": rpc error: code = NotFound desc = could not find container \"b3def3b3258d5d3b3139778d99b11fc801fd4b2db07efc6fc17b8e55787305f0\": container with ID starting with b3def3b3258d5d3b3139778d99b11fc801fd4b2db07efc6fc17b8e55787305f0 not found: ID does not exist"
Sep 05 20:10:24.813910 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:24.813884    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf"]
Sep 05 20:10:24.819454 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:24.819432    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dbbedd-76c9cbb5f9-69nbf"]
Sep 05 20:10:24.833123 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:24.833087    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f30684-3a08-40db-b409-c76c14085fa9-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:24.833220 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:24.833141    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q75qw\" (UniqueName: \"kubernetes.io/projected/71f30684-3a08-40db-b409-c76c14085fa9-kube-api-access-q75qw\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:25.435438 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:25.435401    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c6c77af-2ef1-4839-8093-e8c04478274f" path="/var/lib/kubelet/pods/3c6c77af-2ef1-4839-8093-e8c04478274f/volumes"
Sep 05 20:10:25.435808 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:25.435761    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71f30684-3a08-40db-b409-c76c14085fa9" path="/var/lib/kubelet/pods/71f30684-3a08-40db-b409-c76c14085fa9/volumes"
Sep 05 20:10:31.529609 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:31.529576    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"]
Sep 05 20:10:31.530071 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:31.529771    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5" podUID="9626662f-1234-4008-be31-7627e910a08d" containerName="istio-proxy" containerID="cri-o://3e1a410ab6a5663446ff21c4354676b73d8f454e3abb7df7746b3650f2bb3b6f" gracePeriod=30
Sep 05 20:10:31.868790 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:31.868757    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1436b9-47hfb"]
Sep 05 20:10:31.869383 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:31.869361    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71f30684-3a08-40db-b409-c76c14085fa9" containerName="payload-processing"
Sep 05 20:10:31.869383 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:31.869384    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="71f30684-3a08-40db-b409-c76c14085fa9" containerName="payload-processing"
Sep 05 20:10:31.869563 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:31.869436    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c6c77af-2ef1-4839-8093-e8c04478274f" containerName="revoke-keys"
Sep 05 20:10:31.869563 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:31.869446    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c6c77af-2ef1-4839-8093-e8c04478274f" containerName="revoke-keys"
Sep 05 20:10:31.869670 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:31.869571    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="71f30684-3a08-40db-b409-c76c14085fa9" containerName="payload-processing"
Sep 05 20:10:31.869670 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:31.869587    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c6c77af-2ef1-4839-8093-e8c04478274f" containerName="revoke-keys"
Sep 05 20:10:31.874530 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:31.874509    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1436b9-47hfb"
Sep 05 20:10:31.876816 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:31.876791    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 20:10:31.876934 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:31.876893    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zx89\""
Sep 05 20:10:31.884976 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:31.884950    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1436b9-47hfb"]
Sep 05 20:10:32.003224 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:32.003183    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9d0261-9e27-4dea-8e34-954ca7412de7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1436b9-47hfb\" (UID: \"9a9d0261-9e27-4dea-8e34-954ca7412de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1436b9-47hfb"
Sep 05 20:10:32.104301 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:32.104262    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9d0261-9e27-4dea-8e34-954ca7412de7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1436b9-47hfb\" (UID: \"9a9d0261-9e27-4dea-8e34-954ca7412de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1436b9-47hfb"
Sep 05 20:10:32.104971 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:32.104952    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9d0261-9e27-4dea-8e34-954ca7412de7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1436b9-47hfb\" (UID: \"9a9d0261-9e27-4dea-8e34-954ca7412de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1436b9-47hfb"
Sep 05 20:10:32.188392 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:32.188278    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1436b9-47hfb"
Sep 05 20:10:32.299042 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:32.298990    2583 patch_prober.go:28] interesting pod/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 05 20:10:32.299277 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:32.299086    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5" podUID="9626662f-1234-4008-be31-7627e910a08d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 05 20:10:32.472864 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:32.469817    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1436b9-47hfb"]
Sep 05 20:10:32.842946 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:32.842903    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1436b9-47hfb" event={"ID":"9a9d0261-9e27-4dea-8e34-954ca7412de7","Type":"ContainerStarted","Data":"ab4c75a2353ab6ce330340743ef4b65bd01807f6ca6ea05cc0b47e8cf57e57e2"}
Sep 05 20:10:32.843344 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:32.842968    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1436b9-47hfb" event={"ID":"9a9d0261-9e27-4dea-8e34-954ca7412de7","Type":"ContainerStarted","Data":"5f858082bb53e4baa839c8083d7a9ccdb0420eab1bea75c7ace2e5c14a8dcc2c"}
Sep 05 20:10:32.864788 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:32.864728    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1436b9-47hfb" podStartSLOduration=1.864708109 podStartE2EDuration="1.864708109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:10:32.86232785 +0000 UTC m=+1631.891959755" watchObservedRunningTime="2026-09-05 20:10:32.864708109 +0000 UTC m=+1631.894340019"
Sep 05 20:10:33.853246 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:33.853204    2583 generic.go:356] "Generic (PLEG): container finished" podID="9a9d0261-9e27-4dea-8e34-954ca7412de7" containerID="ab4c75a2353ab6ce330340743ef4b65bd01807f6ca6ea05cc0b47e8cf57e57e2" exitCode=0
Sep 05 20:10:33.853642 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:33.853284    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1436b9-47hfb" event={"ID":"9a9d0261-9e27-4dea-8e34-954ca7412de7","Type":"ContainerDied","Data":"ab4c75a2353ab6ce330340743ef4b65bd01807f6ca6ea05cc0b47e8cf57e57e2"}
Sep 05 20:10:34.997991 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:34.997964    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1436b9-47hfb"
Sep 05 20:10:35.133137 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:35.133023    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9d0261-9e27-4dea-8e34-954ca7412de7-maas-api-service-ca\") pod \"9a9d0261-9e27-4dea-8e34-954ca7412de7\" (UID: \"9a9d0261-9e27-4dea-8e34-954ca7412de7\") "
Sep 05 20:10:35.133399 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:35.133367    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a9d0261-9e27-4dea-8e34-954ca7412de7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9a9d0261-9e27-4dea-8e34-954ca7412de7" (UID: "9a9d0261-9e27-4dea-8e34-954ca7412de7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:10:35.234543 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:35.234512    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a9d0261-9e27-4dea-8e34-954ca7412de7-maas-api-service-ca\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:35.866812 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:35.866779    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1436b9-47hfb"
Sep 05 20:10:35.866812 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:35.866792    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1436b9-47hfb" event={"ID":"9a9d0261-9e27-4dea-8e34-954ca7412de7","Type":"ContainerDied","Data":"5f858082bb53e4baa839c8083d7a9ccdb0420eab1bea75c7ace2e5c14a8dcc2c"}
Sep 05 20:10:35.867063 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:35.866848    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f858082bb53e4baa839c8083d7a9ccdb0420eab1bea75c7ace2e5c14a8dcc2c"
Sep 05 20:10:36.219672 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.219586    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"]
Sep 05 20:10:36.808471 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.808445    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:10:36.883603 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.883566    2583 generic.go:356] "Generic (PLEG): container finished" podID="9626662f-1234-4008-be31-7627e910a08d" containerID="3e1a410ab6a5663446ff21c4354676b73d8f454e3abb7df7746b3650f2bb3b6f" exitCode=0
Sep 05 20:10:36.883773 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.883632    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"
Sep 05 20:10:36.883773 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.883656    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5" event={"ID":"9626662f-1234-4008-be31-7627e910a08d","Type":"ContainerDied","Data":"3e1a410ab6a5663446ff21c4354676b73d8f454e3abb7df7746b3650f2bb3b6f"}
Sep 05 20:10:36.883773 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.883736    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5" event={"ID":"9626662f-1234-4008-be31-7627e910a08d","Type":"ContainerDied","Data":"e482f34a788a3b010ae7224a68a4f958285509c800ac4b58646d4ca6d28811b0"}
Sep 05 20:10:36.883773 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.883758    2583 scope.go:117] "RemoveContainer" containerID="3e1a410ab6a5663446ff21c4354676b73d8f454e3abb7df7746b3650f2bb3b6f"
Sep 05 20:10:36.893681 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.893493    2583 scope.go:117] "RemoveContainer" containerID="e3511647a03be28d9b0d0ecf4001c00275988d583daf209ce849c90f7f55ec83"
Sep 05 20:10:36.902668 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.902647    2583 scope.go:117] "RemoveContainer" containerID="3e1a410ab6a5663446ff21c4354676b73d8f454e3abb7df7746b3650f2bb3b6f"
Sep 05 20:10:36.902952 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:10:36.902934    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e1a410ab6a5663446ff21c4354676b73d8f454e3abb7df7746b3650f2bb3b6f\": container with ID starting with 3e1a410ab6a5663446ff21c4354676b73d8f454e3abb7df7746b3650f2bb3b6f not found: ID does not exist" containerID="3e1a410ab6a5663446ff21c4354676b73d8f454e3abb7df7746b3650f2bb3b6f"
Sep 05 20:10:36.903006 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.902964    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e1a410ab6a5663446ff21c4354676b73d8f454e3abb7df7746b3650f2bb3b6f"} err="failed to get container status \"3e1a410ab6a5663446ff21c4354676b73d8f454e3abb7df7746b3650f2bb3b6f\": rpc error: code = NotFound desc = could not find container \"3e1a410ab6a5663446ff21c4354676b73d8f454e3abb7df7746b3650f2bb3b6f\": container with ID starting with 3e1a410ab6a5663446ff21c4354676b73d8f454e3abb7df7746b3650f2bb3b6f not found: ID does not exist"
Sep 05 20:10:36.903006 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.902981    2583 scope.go:117] "RemoveContainer" containerID="e3511647a03be28d9b0d0ecf4001c00275988d583daf209ce849c90f7f55ec83"
Sep 05 20:10:36.903278 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:10:36.903264    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3511647a03be28d9b0d0ecf4001c00275988d583daf209ce849c90f7f55ec83\": container with ID starting with e3511647a03be28d9b0d0ecf4001c00275988d583daf209ce849c90f7f55ec83 not found: ID does not exist" containerID="e3511647a03be28d9b0d0ecf4001c00275988d583daf209ce849c90f7f55ec83"
Sep 05 20:10:36.903322 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.903290    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3511647a03be28d9b0d0ecf4001c00275988d583daf209ce849c90f7f55ec83"} err="failed to get container status \"e3511647a03be28d9b0d0ecf4001c00275988d583daf209ce849c90f7f55ec83\": rpc error: code = NotFound desc = could not find container \"e3511647a03be28d9b0d0ecf4001c00275988d583daf209ce849c90f7f55ec83\": container with ID starting with e3511647a03be28d9b0d0ecf4001c00275988d583daf209ce849c90f7f55ec83 not found: ID does not exist"
Sep 05 20:10:36.949884 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.949850    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-istio-data\") pod \"9626662f-1234-4008-be31-7627e910a08d\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") "
Sep 05 20:10:36.949884 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.949888    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9626662f-1234-4008-be31-7627e910a08d-istiod-ca-cert\") pod \"9626662f-1234-4008-be31-7627e910a08d\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") "
Sep 05 20:10:36.949884 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.949906    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-credential-socket\") pod \"9626662f-1234-4008-be31-7627e910a08d\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") "
Sep 05 20:10:36.950222 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.949930    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-istio-envoy\") pod \"9626662f-1234-4008-be31-7627e910a08d\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") "
Sep 05 20:10:36.950222 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.949970    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-workload-certs\") pod \"9626662f-1234-4008-be31-7627e910a08d\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") "
Sep 05 20:10:36.950222 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.950002    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9626662f-1234-4008-be31-7627e910a08d-istio-podinfo\") pod \"9626662f-1234-4008-be31-7627e910a08d\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") "
Sep 05 20:10:36.950222 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.950061    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4gmm\" (UniqueName: \"kubernetes.io/projected/9626662f-1234-4008-be31-7627e910a08d-kube-api-access-n4gmm\") pod \"9626662f-1234-4008-be31-7627e910a08d\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") "
Sep 05 20:10:36.950222 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.950148    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-workload-socket\") pod \"9626662f-1234-4008-be31-7627e910a08d\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") "
Sep 05 20:10:36.950222 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.950207    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9626662f-1234-4008-be31-7627e910a08d-istio-token\") pod \"9626662f-1234-4008-be31-7627e910a08d\" (UID: \"9626662f-1234-4008-be31-7627e910a08d\") "
Sep 05 20:10:36.950523 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.950259    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-istio-data" (OuterVolumeSpecName: "istio-data") pod "9626662f-1234-4008-be31-7627e910a08d" (UID: "9626662f-1234-4008-be31-7627e910a08d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:10:36.950523 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.950339    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9626662f-1234-4008-be31-7627e910a08d" (UID: "9626662f-1234-4008-be31-7627e910a08d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:10:36.950523 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.950479    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9626662f-1234-4008-be31-7627e910a08d" (UID: "9626662f-1234-4008-be31-7627e910a08d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:10:36.950698 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.950549    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9626662f-1234-4008-be31-7627e910a08d" (UID: "9626662f-1234-4008-be31-7627e910a08d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:10:36.950698 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.950634    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-istio-data\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:36.950698 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.950655    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-workload-certs\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:36.950698 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.950667    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-workload-socket\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:36.950917 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.950808    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9626662f-1234-4008-be31-7627e910a08d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9626662f-1234-4008-be31-7627e910a08d" (UID: "9626662f-1234-4008-be31-7627e910a08d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:10:36.952550 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.952514    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9626662f-1234-4008-be31-7627e910a08d-istio-token" (OuterVolumeSpecName: "istio-token") pod "9626662f-1234-4008-be31-7627e910a08d" (UID: "9626662f-1234-4008-be31-7627e910a08d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:10:36.952550 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.952530    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9626662f-1234-4008-be31-7627e910a08d-kube-api-access-n4gmm" (OuterVolumeSpecName: "kube-api-access-n4gmm") pod "9626662f-1234-4008-be31-7627e910a08d" (UID: "9626662f-1234-4008-be31-7627e910a08d"). InnerVolumeSpecName "kube-api-access-n4gmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:10:36.952755 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.952580    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9626662f-1234-4008-be31-7627e910a08d" (UID: "9626662f-1234-4008-be31-7627e910a08d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:10:36.952930 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:36.952902    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9626662f-1234-4008-be31-7627e910a08d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9626662f-1234-4008-be31-7627e910a08d" (UID: "9626662f-1234-4008-be31-7627e910a08d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 20:10:37.051492 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:37.051406    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9626662f-1234-4008-be31-7627e910a08d-istio-token\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:37.051492 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:37.051436    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9626662f-1234-4008-be31-7627e910a08d-istiod-ca-cert\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:37.051492 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:37.051445    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-credential-socket\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:37.051492 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:37.051454    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9626662f-1234-4008-be31-7627e910a08d-istio-envoy\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:37.051492 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:37.051463    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9626662f-1234-4008-be31-7627e910a08d-istio-podinfo\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:37.051492 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:37.051472    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4gmm\" (UniqueName: \"kubernetes.io/projected/9626662f-1234-4008-be31-7627e910a08d-kube-api-access-n4gmm\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:37.209576 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:37.209542    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"]
Sep 05 20:10:37.213852 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:37.213822    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-dbbedd-openshift-default-8559b65d48-c42t5"]
Sep 05 20:10:37.435747 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:37.435705    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9626662f-1234-4008-be31-7627e910a08d" path="/var/lib/kubelet/pods/9626662f-1234-4008-be31-7627e910a08d/volumes"
Sep 05 20:10:41.707458 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.707418    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd"]
Sep 05 20:10:41.708042 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.708022    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a9d0261-9e27-4dea-8e34-954ca7412de7" containerName="revoke-keys"
Sep 05 20:10:41.708088 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.708047    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a9d0261-9e27-4dea-8e34-954ca7412de7" containerName="revoke-keys"
Sep 05 20:10:41.708088 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.708069    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9626662f-1234-4008-be31-7627e910a08d" containerName="istio-proxy"
Sep 05 20:10:41.708088 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.708078    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9626662f-1234-4008-be31-7627e910a08d" containerName="istio-proxy"
Sep 05 20:10:41.708222 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.708201    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a9d0261-9e27-4dea-8e34-954ca7412de7" containerName="revoke-keys"
Sep 05 20:10:41.708222 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.708212    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9626662f-1234-4008-be31-7627e910a08d" containerName="istio-proxy"
Sep 05 20:10:41.708222 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.708219    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9626662f-1234-4008-be31-7627e910a08d" containerName="istio-proxy"
Sep 05 20:10:41.713210 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.713184    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd"
Sep 05 20:10:41.715871 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.715842    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz"]
Sep 05 20:10:41.716071 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.716049    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d0f6b5\""
Sep 05 20:10:41.716525 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.716503    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d0f6b5-dockercfg-q98t2\""
Sep 05 20:10:41.716828 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.716806    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9626662f-1234-4008-be31-7627e910a08d" containerName="istio-proxy"
Sep 05 20:10:41.716907 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.716832    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9626662f-1234-4008-be31-7627e910a08d" containerName="istio-proxy"
Sep 05 20:10:41.721178 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.720993    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz"
Sep 05 20:10:41.721596 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.721576    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd"]
Sep 05 20:10:41.731716 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.731685    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz"]
Sep 05 20:10:41.791508 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.791460    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99c1d1ac-b6c8-4047-83e8-168b8b94fc56-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz\" (UID: \"99c1d1ac-b6c8-4047-83e8-168b8b94fc56\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz"
Sep 05 20:10:41.791672 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.791514    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2zqw\" (UniqueName: \"kubernetes.io/projected/99c1d1ac-b6c8-4047-83e8-168b8b94fc56-kube-api-access-s2zqw\") pod \"payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz\" (UID: \"99c1d1ac-b6c8-4047-83e8-168b8b94fc56\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz"
Sep 05 20:10:41.791672 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.791576    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067dea99-d8a2-4a9e-a804-f24b6d5421e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd\" (UID: \"067dea99-d8a2-4a9e-a804-f24b6d5421e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd"
Sep 05 20:10:41.791672 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.791603    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z5t5\" (UniqueName: \"kubernetes.io/projected/067dea99-d8a2-4a9e-a804-f24b6d5421e2-kube-api-access-8z5t5\") pod \"payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd\" (UID: \"067dea99-d8a2-4a9e-a804-f24b6d5421e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd"
Sep 05 20:10:41.892481 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.892452    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99c1d1ac-b6c8-4047-83e8-168b8b94fc56-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz\" (UID: \"99c1d1ac-b6c8-4047-83e8-168b8b94fc56\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz"
Sep 05 20:10:41.892686 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.892488    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2zqw\" (UniqueName: \"kubernetes.io/projected/99c1d1ac-b6c8-4047-83e8-168b8b94fc56-kube-api-access-s2zqw\") pod \"payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz\" (UID: \"99c1d1ac-b6c8-4047-83e8-168b8b94fc56\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz"
Sep 05 20:10:41.892686 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.892523    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067dea99-d8a2-4a9e-a804-f24b6d5421e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd\" (UID: \"067dea99-d8a2-4a9e-a804-f24b6d5421e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd"
Sep 05 20:10:41.892686 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.892546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z5t5\" (UniqueName: \"kubernetes.io/projected/067dea99-d8a2-4a9e-a804-f24b6d5421e2-kube-api-access-8z5t5\") pod \"payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd\" (UID: \"067dea99-d8a2-4a9e-a804-f24b6d5421e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd"
Sep 05 20:10:41.893284 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.893258    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067dea99-d8a2-4a9e-a804-f24b6d5421e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd\" (UID: \"067dea99-d8a2-4a9e-a804-f24b6d5421e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd"
Sep 05 20:10:41.893427 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.893407    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99c1d1ac-b6c8-4047-83e8-168b8b94fc56-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz\" (UID: \"99c1d1ac-b6c8-4047-83e8-168b8b94fc56\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz"
Sep 05 20:10:41.902463 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.902434    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z5t5\" (UniqueName: \"kubernetes.io/projected/067dea99-d8a2-4a9e-a804-f24b6d5421e2-kube-api-access-8z5t5\") pod \"payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd\" (UID: \"067dea99-d8a2-4a9e-a804-f24b6d5421e2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd"
Sep 05 20:10:41.902594 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:41.902546    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2zqw\" (UniqueName: \"kubernetes.io/projected/99c1d1ac-b6c8-4047-83e8-168b8b94fc56-kube-api-access-s2zqw\") pod \"payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz\" (UID: \"99c1d1ac-b6c8-4047-83e8-168b8b94fc56\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz"
Sep 05 20:10:42.027074 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.026976    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd"
Sep 05 20:10:42.037200 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.037169    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz"
Sep 05 20:10:42.312439 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.312404    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd"]
Sep 05 20:10:42.313585 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:10:42.313557    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067dea99_d8a2_4a9e_a804_f24b6d5421e2.slice/crio-dcd07d94c820dac2bab1e69e38bad516a58d5a671082b4970f62d92c78e522cd WatchSource:0}: Error finding container dcd07d94c820dac2bab1e69e38bad516a58d5a671082b4970f62d92c78e522cd: Status 404 returned error can't find the container with id dcd07d94c820dac2bab1e69e38bad516a58d5a671082b4970f62d92c78e522cd
Sep 05 20:10:42.335124 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.334309    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz"]
Sep 05 20:10:42.416742 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.416702    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj"]
Sep 05 20:10:42.417617 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.417223    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj" podUID="6fa28148-f4e9-488f-89fd-4cf9051c538b" containerName="payload-pre-processing" containerID="cri-o://a48509fd04e5c65e42c11bcabc0ae2d6cda5fdb270bdea8ac485d6d96b5bfd68" gracePeriod=30
Sep 05 20:10:42.586508 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.586452    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1436b9-47hfb"]
Sep 05 20:10:42.590559 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.590462    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1436b9-47hfb"]
Sep 05 20:10:42.681495 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.681467    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj"
Sep 05 20:10:42.702390 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.701164    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq7j2\" (UniqueName: \"kubernetes.io/projected/6fa28148-f4e9-488f-89fd-4cf9051c538b-kube-api-access-mq7j2\") pod \"6fa28148-f4e9-488f-89fd-4cf9051c538b\" (UID: \"6fa28148-f4e9-488f-89fd-4cf9051c538b\") "
Sep 05 20:10:42.702390 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.701253    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa28148-f4e9-488f-89fd-4cf9051c538b-plugins-config-volume\") pod \"6fa28148-f4e9-488f-89fd-4cf9051c538b\" (UID: \"6fa28148-f4e9-488f-89fd-4cf9051c538b\") "
Sep 05 20:10:42.702390 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.701609    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fa28148-f4e9-488f-89fd-4cf9051c538b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6fa28148-f4e9-488f-89fd-4cf9051c538b" (UID: "6fa28148-f4e9-488f-89fd-4cf9051c538b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:10:42.702390 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.701972    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa28148-f4e9-488f-89fd-4cf9051c538b-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:42.704123 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.704072    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa28148-f4e9-488f-89fd-4cf9051c538b-kube-api-access-mq7j2" (OuterVolumeSpecName: "kube-api-access-mq7j2") pod "6fa28148-f4e9-488f-89fd-4cf9051c538b" (UID: "6fa28148-f4e9-488f-89fd-4cf9051c538b"). InnerVolumeSpecName "kube-api-access-mq7j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:10:42.802766 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.802727    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq7j2\" (UniqueName: \"kubernetes.io/projected/6fa28148-f4e9-488f-89fd-4cf9051c538b-kube-api-access-mq7j2\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:42.923546 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.923506    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz" event={"ID":"99c1d1ac-b6c8-4047-83e8-168b8b94fc56","Type":"ContainerStarted","Data":"584de74700831f0c9965770e22182644db6b52f44f4177c39f41e24825ce5c8a"}
Sep 05 20:10:42.923739 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.923572    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz" event={"ID":"99c1d1ac-b6c8-4047-83e8-168b8b94fc56","Type":"ContainerStarted","Data":"310de22a5d47eb63e595eded2ae89393675bf399b7636eba2b3a47f24356772f"}
Sep 05 20:10:42.923739 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.923719    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz"
Sep 05 20:10:42.925263 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.925234    2583 generic.go:356] "Generic (PLEG): container finished" podID="6fa28148-f4e9-488f-89fd-4cf9051c538b" containerID="a48509fd04e5c65e42c11bcabc0ae2d6cda5fdb270bdea8ac485d6d96b5bfd68" exitCode=0
Sep 05 20:10:42.925469 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.925279    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj"
Sep 05 20:10:42.925469 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.925370    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj" event={"ID":"6fa28148-f4e9-488f-89fd-4cf9051c538b","Type":"ContainerDied","Data":"a48509fd04e5c65e42c11bcabc0ae2d6cda5fdb270bdea8ac485d6d96b5bfd68"}
Sep 05 20:10:42.925469 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.925405    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj" event={"ID":"6fa28148-f4e9-488f-89fd-4cf9051c538b","Type":"ContainerDied","Data":"1bc99a066d5e7d2e5f17313dc320554773c1d7a123032e3bed1567be7da281c2"}
Sep 05 20:10:42.925469 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.925428    2583 scope.go:117] "RemoveContainer" containerID="a48509fd04e5c65e42c11bcabc0ae2d6cda5fdb270bdea8ac485d6d96b5bfd68"
Sep 05 20:10:42.927337 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.927313    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd" event={"ID":"067dea99-d8a2-4a9e-a804-f24b6d5421e2","Type":"ContainerStarted","Data":"a5d85a13c75e7a81d2144db48e1a2561b38256336e73f4613a6e637fa6fa98a2"}
Sep 05 20:10:42.927455 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.927366    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd" event={"ID":"067dea99-d8a2-4a9e-a804-f24b6d5421e2","Type":"ContainerStarted","Data":"dcd07d94c820dac2bab1e69e38bad516a58d5a671082b4970f62d92c78e522cd"}
Sep 05 20:10:42.927455 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.927391    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd"
Sep 05 20:10:42.936312 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.936296    2583 scope.go:117] "RemoveContainer" containerID="a48509fd04e5c65e42c11bcabc0ae2d6cda5fdb270bdea8ac485d6d96b5bfd68"
Sep 05 20:10:42.936568 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:10:42.936552    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a48509fd04e5c65e42c11bcabc0ae2d6cda5fdb270bdea8ac485d6d96b5bfd68\": container with ID starting with a48509fd04e5c65e42c11bcabc0ae2d6cda5fdb270bdea8ac485d6d96b5bfd68 not found: ID does not exist" containerID="a48509fd04e5c65e42c11bcabc0ae2d6cda5fdb270bdea8ac485d6d96b5bfd68"
Sep 05 20:10:42.936625 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.936576    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a48509fd04e5c65e42c11bcabc0ae2d6cda5fdb270bdea8ac485d6d96b5bfd68"} err="failed to get container status \"a48509fd04e5c65e42c11bcabc0ae2d6cda5fdb270bdea8ac485d6d96b5bfd68\": rpc error: code = NotFound desc = could not find container \"a48509fd04e5c65e42c11bcabc0ae2d6cda5fdb270bdea8ac485d6d96b5bfd68\": container with ID starting with a48509fd04e5c65e42c11bcabc0ae2d6cda5fdb270bdea8ac485d6d96b5bfd68 not found: ID does not exist"
Sep 05 20:10:42.946266 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.946221    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz" podStartSLOduration=1.946208765 podStartE2EDuration="1.946208765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:10:42.943671567 +0000 UTC m=+1641.973303471" watchObservedRunningTime="2026-09-05 20:10:42.946208765 +0000 UTC m=+1641.975840669"
Sep 05 20:10:42.961470 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.961443    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj"]
Sep 05 20:10:42.963942 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.963923    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1436b9-5765f47bc4-gq6fj"]
Sep 05 20:10:42.980753 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:42.980698    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd" podStartSLOduration=1.980684255 podStartE2EDuration="1.980684255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:10:42.977342263 +0000 UTC m=+1642.006974168" watchObservedRunningTime="2026-09-05 20:10:42.980684255 +0000 UTC m=+1642.010316160"
Sep 05 20:10:43.435420 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:43.435386    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fa28148-f4e9-488f-89fd-4cf9051c538b" path="/var/lib/kubelet/pods/6fa28148-f4e9-488f-89fd-4cf9051c538b/volumes"
Sep 05 20:10:43.435731 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:43.435719    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a9d0261-9e27-4dea-8e34-954ca7412de7" path="/var/lib/kubelet/pods/9a9d0261-9e27-4dea-8e34-954ca7412de7/volumes"
Sep 05 20:10:49.491904 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:49.491868    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca4fba-xm4xw"]
Sep 05 20:10:49.492333 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:49.492316    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fa28148-f4e9-488f-89fd-4cf9051c538b" containerName="payload-pre-processing"
Sep 05 20:10:49.492333 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:49.492328    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fa28148-f4e9-488f-89fd-4cf9051c538b" containerName="payload-pre-processing"
Sep 05 20:10:49.492431 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:49.492418    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fa28148-f4e9-488f-89fd-4cf9051c538b" containerName="payload-pre-processing"
Sep 05 20:10:49.496944 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:49.496925    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca4fba-xm4xw"
Sep 05 20:10:49.499427 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:49.499402    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 20:10:49.500035 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:49.500013    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zx89\""
Sep 05 20:10:49.503950 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:49.503928    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca4fba-xm4xw"]
Sep 05 20:10:49.563764 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:49.563727    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0672873f-59c9-4f37-9a24-09cdc11f2abc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ca4fba-xm4xw\" (UID: \"0672873f-59c9-4f37-9a24-09cdc11f2abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca4fba-xm4xw"
Sep 05 20:10:49.664494 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:49.664449    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0672873f-59c9-4f37-9a24-09cdc11f2abc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ca4fba-xm4xw\" (UID: \"0672873f-59c9-4f37-9a24-09cdc11f2abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca4fba-xm4xw"
Sep 05 20:10:49.665170 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:49.665143    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0672873f-59c9-4f37-9a24-09cdc11f2abc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ca4fba-xm4xw\" (UID: \"0672873f-59c9-4f37-9a24-09cdc11f2abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca4fba-xm4xw"
Sep 05 20:10:49.809459 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:49.809358    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca4fba-xm4xw"
Sep 05 20:10:50.098782 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:50.098750    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca4fba-xm4xw"]
Sep 05 20:10:50.100987 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:10:50.100955    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0672873f_59c9_4f37_9a24_09cdc11f2abc.slice/crio-aab5983ec4291056c157d8bc74b7bdcfd5185829e9b54e2a5b8755d785592101 WatchSource:0}: Error finding container aab5983ec4291056c157d8bc74b7bdcfd5185829e9b54e2a5b8755d785592101: Status 404 returned error can't find the container with id aab5983ec4291056c157d8bc74b7bdcfd5185829e9b54e2a5b8755d785592101
Sep 05 20:10:50.980102 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:50.980076    2583 generic.go:356] "Generic (PLEG): container finished" podID="0672873f-59c9-4f37-9a24-09cdc11f2abc" containerID="1539dc7b80e9edca29b448c454be7c6cdc2e0db2842408a56bc9916e3caa3365" exitCode=0
Sep 05 20:10:50.980429 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:50.980175    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca4fba-xm4xw" event={"ID":"0672873f-59c9-4f37-9a24-09cdc11f2abc","Type":"ContainerDied","Data":"1539dc7b80e9edca29b448c454be7c6cdc2e0db2842408a56bc9916e3caa3365"}
Sep 05 20:10:50.980429 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:50.980259    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca4fba-xm4xw" event={"ID":"0672873f-59c9-4f37-9a24-09cdc11f2abc","Type":"ContainerStarted","Data":"aab5983ec4291056c157d8bc74b7bdcfd5185829e9b54e2a5b8755d785592101"}
Sep 05 20:10:51.239858 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.239756    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd" podUID="1bec1fad-754c-48c4-bba6-b9fc839a1709" containerName="main" containerID="cri-o://66dc1e89e58dd50cf0f64845af1cdcd0085b1e1f7f465a19b12198f639b0262c" gracePeriod=45
Sep 05 20:10:51.499138 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.499048    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:10:51.580651 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.580600    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bec1fad-754c-48c4-bba6-b9fc839a1709-tmp-dir\") pod \"1bec1fad-754c-48c4-bba6-b9fc839a1709\" (UID: \"1bec1fad-754c-48c4-bba6-b9fc839a1709\") "
Sep 05 20:10:51.580849 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.580686    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9dxv\" (UniqueName: \"kubernetes.io/projected/1bec1fad-754c-48c4-bba6-b9fc839a1709-kube-api-access-r9dxv\") pod \"1bec1fad-754c-48c4-bba6-b9fc839a1709\" (UID: \"1bec1fad-754c-48c4-bba6-b9fc839a1709\") "
Sep 05 20:10:51.580849 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.580736    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bec1fad-754c-48c4-bba6-b9fc839a1709-home\") pod \"1bec1fad-754c-48c4-bba6-b9fc839a1709\" (UID: \"1bec1fad-754c-48c4-bba6-b9fc839a1709\") "
Sep 05 20:10:51.580849 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.580783    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bec1fad-754c-48c4-bba6-b9fc839a1709-tls-certs\") pod \"1bec1fad-754c-48c4-bba6-b9fc839a1709\" (UID: \"1bec1fad-754c-48c4-bba6-b9fc839a1709\") "
Sep 05 20:10:51.580849 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.580809    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bec1fad-754c-48c4-bba6-b9fc839a1709-model-cache\") pod \"1bec1fad-754c-48c4-bba6-b9fc839a1709\" (UID: \"1bec1fad-754c-48c4-bba6-b9fc839a1709\") "
Sep 05 20:10:51.580849 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.580844    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bec1fad-754c-48c4-bba6-b9fc839a1709-dshm\") pod \"1bec1fad-754c-48c4-bba6-b9fc839a1709\" (UID: \"1bec1fad-754c-48c4-bba6-b9fc839a1709\") "
Sep 05 20:10:51.581131 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.580876    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bec1fad-754c-48c4-bba6-b9fc839a1709-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1bec1fad-754c-48c4-bba6-b9fc839a1709" (UID: "1bec1fad-754c-48c4-bba6-b9fc839a1709"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:10:51.581131 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.581069    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bec1fad-754c-48c4-bba6-b9fc839a1709-home" (OuterVolumeSpecName: "home") pod "1bec1fad-754c-48c4-bba6-b9fc839a1709" (UID: "1bec1fad-754c-48c4-bba6-b9fc839a1709"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:10:51.581131 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.581084    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bec1fad-754c-48c4-bba6-b9fc839a1709-model-cache" (OuterVolumeSpecName: "model-cache") pod "1bec1fad-754c-48c4-bba6-b9fc839a1709" (UID: "1bec1fad-754c-48c4-bba6-b9fc839a1709"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:10:51.581297 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.581186    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bec1fad-754c-48c4-bba6-b9fc839a1709-tmp-dir\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:51.581297 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.581201    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bec1fad-754c-48c4-bba6-b9fc839a1709-home\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:51.581297 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.581209    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bec1fad-754c-48c4-bba6-b9fc839a1709-model-cache\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:51.582890 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.582859    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bec1fad-754c-48c4-bba6-b9fc839a1709-dshm" (OuterVolumeSpecName: "dshm") pod "1bec1fad-754c-48c4-bba6-b9fc839a1709" (UID: "1bec1fad-754c-48c4-bba6-b9fc839a1709"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:10:51.582890 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.582881    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bec1fad-754c-48c4-bba6-b9fc839a1709-kube-api-access-r9dxv" (OuterVolumeSpecName: "kube-api-access-r9dxv") pod "1bec1fad-754c-48c4-bba6-b9fc839a1709" (UID: "1bec1fad-754c-48c4-bba6-b9fc839a1709"). InnerVolumeSpecName "kube-api-access-r9dxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:10:51.583048 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.582907    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bec1fad-754c-48c4-bba6-b9fc839a1709-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1bec1fad-754c-48c4-bba6-b9fc839a1709" (UID: "1bec1fad-754c-48c4-bba6-b9fc839a1709"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:10:51.682338 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.682289    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9dxv\" (UniqueName: \"kubernetes.io/projected/1bec1fad-754c-48c4-bba6-b9fc839a1709-kube-api-access-r9dxv\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:51.682338 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.682334    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bec1fad-754c-48c4-bba6-b9fc839a1709-tls-certs\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:51.682338 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.682345    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bec1fad-754c-48c4-bba6-b9fc839a1709-dshm\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:51.990602 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.990564    2583 generic.go:356] "Generic (PLEG): container finished" podID="1bec1fad-754c-48c4-bba6-b9fc839a1709" containerID="66dc1e89e58dd50cf0f64845af1cdcd0085b1e1f7f465a19b12198f639b0262c" exitCode=0
Sep 05 20:10:51.991103 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.990607    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"
Sep 05 20:10:51.991103 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.990643    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd" event={"ID":"1bec1fad-754c-48c4-bba6-b9fc839a1709","Type":"ContainerDied","Data":"66dc1e89e58dd50cf0f64845af1cdcd0085b1e1f7f465a19b12198f639b0262c"}
Sep 05 20:10:51.991103 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.990700    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd" event={"ID":"1bec1fad-754c-48c4-bba6-b9fc839a1709","Type":"ContainerDied","Data":"4527e77f792e60a8b682055e0d50bbd0b0ffe57354ef2a3c465db28f2aea8013"}
Sep 05 20:10:51.991103 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:51.990721    2583 scope.go:117] "RemoveContainer" containerID="66dc1e89e58dd50cf0f64845af1cdcd0085b1e1f7f465a19b12198f639b0262c"
Sep 05 20:10:52.001916 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:52.001895    2583 scope.go:117] "RemoveContainer" containerID="66dc1e89e58dd50cf0f64845af1cdcd0085b1e1f7f465a19b12198f639b0262c"
Sep 05 20:10:52.002244 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:10:52.002224    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66dc1e89e58dd50cf0f64845af1cdcd0085b1e1f7f465a19b12198f639b0262c\": container with ID starting with 66dc1e89e58dd50cf0f64845af1cdcd0085b1e1f7f465a19b12198f639b0262c not found: ID does not exist" containerID="66dc1e89e58dd50cf0f64845af1cdcd0085b1e1f7f465a19b12198f639b0262c"
Sep 05 20:10:52.002321 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:52.002253    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66dc1e89e58dd50cf0f64845af1cdcd0085b1e1f7f465a19b12198f639b0262c"} err="failed to get container status \"66dc1e89e58dd50cf0f64845af1cdcd0085b1e1f7f465a19b12198f639b0262c\": rpc error: code = NotFound desc = could not find container \"66dc1e89e58dd50cf0f64845af1cdcd0085b1e1f7f465a19b12198f639b0262c\": container with ID starting with 66dc1e89e58dd50cf0f64845af1cdcd0085b1e1f7f465a19b12198f639b0262c not found: ID does not exist"
Sep 05 20:10:52.014821 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:52.014779    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"]
Sep 05 20:10:52.018533 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:52.018502    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0cc31a/rate-test-model-0cc31a-kserve-668d67fb6c-9zjsd"]
Sep 05 20:10:52.139023 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:52.138998    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca4fba-xm4xw"
Sep 05 20:10:52.186122 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:52.186074    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0672873f-59c9-4f37-9a24-09cdc11f2abc-maas-api-service-ca\") pod \"0672873f-59c9-4f37-9a24-09cdc11f2abc\" (UID: \"0672873f-59c9-4f37-9a24-09cdc11f2abc\") "
Sep 05 20:10:52.186438 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:52.186417    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0672873f-59c9-4f37-9a24-09cdc11f2abc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0672873f-59c9-4f37-9a24-09cdc11f2abc" (UID: "0672873f-59c9-4f37-9a24-09cdc11f2abc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:10:52.287685 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:52.287597    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0672873f-59c9-4f37-9a24-09cdc11f2abc-maas-api-service-ca\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:10:53.000184 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:53.000146    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca4fba-xm4xw" event={"ID":"0672873f-59c9-4f37-9a24-09cdc11f2abc","Type":"ContainerDied","Data":"aab5983ec4291056c157d8bc74b7bdcfd5185829e9b54e2a5b8755d785592101"}
Sep 05 20:10:53.000184 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:53.000169    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca4fba-xm4xw"
Sep 05 20:10:53.000619 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:53.000209    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aab5983ec4291056c157d8bc74b7bdcfd5185829e9b54e2a5b8755d785592101"
Sep 05 20:10:53.435034 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:53.434997    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bec1fad-754c-48c4-bba6-b9fc839a1709" path="/var/lib/kubelet/pods/1bec1fad-754c-48c4-bba6-b9fc839a1709/volumes"
Sep 05 20:10:53.937295 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:53.937266    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd"
Sep 05 20:10:53.937466 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:10:53.937335    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz"
Sep 05 20:11:00.224278 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:00.224236    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4"]
Sep 05 20:11:00.224762 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:00.224521    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4" podUID="d8eb8a4e-6700-4b55-8777-00752b3695b9" containerName="payload-pre-processing" containerID="cri-o://c55b5e31aa1719fdfe96b653aac7bf1195935fa900bf570ee8d74dbbc422bb5b" gracePeriod=30
Sep 05 20:11:00.362427 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:00.362388    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca4fba-xm4xw"]
Sep 05 20:11:00.366574 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:00.366538    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ca4fba-xm4xw"]
Sep 05 20:11:00.502124 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:00.502090    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4"
Sep 05 20:11:00.567619 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:00.567589    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpr6p\" (UniqueName: \"kubernetes.io/projected/d8eb8a4e-6700-4b55-8777-00752b3695b9-kube-api-access-dpr6p\") pod \"d8eb8a4e-6700-4b55-8777-00752b3695b9\" (UID: \"d8eb8a4e-6700-4b55-8777-00752b3695b9\") "
Sep 05 20:11:00.567815 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:00.567661    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8eb8a4e-6700-4b55-8777-00752b3695b9-plugins-config-volume\") pod \"d8eb8a4e-6700-4b55-8777-00752b3695b9\" (UID: \"d8eb8a4e-6700-4b55-8777-00752b3695b9\") "
Sep 05 20:11:00.568201 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:00.568149    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8eb8a4e-6700-4b55-8777-00752b3695b9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d8eb8a4e-6700-4b55-8777-00752b3695b9" (UID: "d8eb8a4e-6700-4b55-8777-00752b3695b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:11:00.569833 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:00.569805    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8eb8a4e-6700-4b55-8777-00752b3695b9-kube-api-access-dpr6p" (OuterVolumeSpecName: "kube-api-access-dpr6p") pod "d8eb8a4e-6700-4b55-8777-00752b3695b9" (UID: "d8eb8a4e-6700-4b55-8777-00752b3695b9"). InnerVolumeSpecName "kube-api-access-dpr6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:11:00.669522 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:00.669485    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpr6p\" (UniqueName: \"kubernetes.io/projected/d8eb8a4e-6700-4b55-8777-00752b3695b9-kube-api-access-dpr6p\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:11:00.669522 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:00.669517    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8eb8a4e-6700-4b55-8777-00752b3695b9-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:11:01.052594 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:01.052557    2583 generic.go:356] "Generic (PLEG): container finished" podID="d8eb8a4e-6700-4b55-8777-00752b3695b9" containerID="c55b5e31aa1719fdfe96b653aac7bf1195935fa900bf570ee8d74dbbc422bb5b" exitCode=0
Sep 05 20:11:01.052746 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:01.052601    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4"
Sep 05 20:11:01.052746 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:01.052636    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4" event={"ID":"d8eb8a4e-6700-4b55-8777-00752b3695b9","Type":"ContainerDied","Data":"c55b5e31aa1719fdfe96b653aac7bf1195935fa900bf570ee8d74dbbc422bb5b"}
Sep 05 20:11:01.052746 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:01.052693    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4" event={"ID":"d8eb8a4e-6700-4b55-8777-00752b3695b9","Type":"ContainerDied","Data":"904d094e39f31150ffc12c075cd3012aa2edf095bca55d8df7ceec5e52a9013b"}
Sep 05 20:11:01.052746 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:01.052716    2583 scope.go:117] "RemoveContainer" containerID="c55b5e31aa1719fdfe96b653aac7bf1195935fa900bf570ee8d74dbbc422bb5b"
Sep 05 20:11:01.062681 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:01.062661    2583 scope.go:117] "RemoveContainer" containerID="c55b5e31aa1719fdfe96b653aac7bf1195935fa900bf570ee8d74dbbc422bb5b"
Sep 05 20:11:01.062981 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:11:01.062961    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c55b5e31aa1719fdfe96b653aac7bf1195935fa900bf570ee8d74dbbc422bb5b\": container with ID starting with c55b5e31aa1719fdfe96b653aac7bf1195935fa900bf570ee8d74dbbc422bb5b not found: ID does not exist" containerID="c55b5e31aa1719fdfe96b653aac7bf1195935fa900bf570ee8d74dbbc422bb5b"
Sep 05 20:11:01.063035 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:01.062991    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c55b5e31aa1719fdfe96b653aac7bf1195935fa900bf570ee8d74dbbc422bb5b"} err="failed to get container status \"c55b5e31aa1719fdfe96b653aac7bf1195935fa900bf570ee8d74dbbc422bb5b\": rpc error: code = NotFound desc = could not find container \"c55b5e31aa1719fdfe96b653aac7bf1195935fa900bf570ee8d74dbbc422bb5b\": container with ID starting with c55b5e31aa1719fdfe96b653aac7bf1195935fa900bf570ee8d74dbbc422bb5b not found: ID does not exist"
Sep 05 20:11:01.074544 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:01.074510    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4"]
Sep 05 20:11:01.080502 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:01.080474    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca4fba-d77d959c8-lqks4"]
Sep 05 20:11:01.437365 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:01.437331    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0672873f-59c9-4f37-9a24-09cdc11f2abc" path="/var/lib/kubelet/pods/0672873f-59c9-4f37-9a24-09cdc11f2abc/volumes"
Sep 05 20:11:01.437868 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:01.437845    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8eb8a4e-6700-4b55-8777-00752b3695b9" path="/var/lib/kubelet/pods/d8eb8a4e-6700-4b55-8777-00752b3695b9/volumes"
Sep 05 20:11:10.900472 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:10.900429    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5"]
Sep 05 20:11:10.901075 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:10.901053    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bec1fad-754c-48c4-bba6-b9fc839a1709" containerName="main"
Sep 05 20:11:10.901183 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:10.901078    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bec1fad-754c-48c4-bba6-b9fc839a1709" containerName="main"
Sep 05 20:11:10.901183 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:10.901120    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0672873f-59c9-4f37-9a24-09cdc11f2abc" containerName="revoke-keys"
Sep 05 20:11:10.901183 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:10.901131    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0672873f-59c9-4f37-9a24-09cdc11f2abc" containerName="revoke-keys"
Sep 05 20:11:10.901183 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:10.901152    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8eb8a4e-6700-4b55-8777-00752b3695b9" containerName="payload-pre-processing"
Sep 05 20:11:10.901183 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:10.901163    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8eb8a4e-6700-4b55-8777-00752b3695b9" containerName="payload-pre-processing"
Sep 05 20:11:10.901433 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:10.901317    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8eb8a4e-6700-4b55-8777-00752b3695b9" containerName="payload-pre-processing"
Sep 05 20:11:10.901433 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:10.901327    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bec1fad-754c-48c4-bba6-b9fc839a1709" containerName="main"
Sep 05 20:11:10.901433 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:10.901340    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0672873f-59c9-4f37-9a24-09cdc11f2abc" containerName="revoke-keys"
Sep 05 20:11:10.906000 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:10.905976    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5"
Sep 05 20:11:10.908240 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:10.908216    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6693ae-dockercfg-g9wsj\""
Sep 05 20:11:10.908808 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:10.908787    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6693ae\""
Sep 05 20:11:10.915332 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:10.915307    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5"]
Sep 05 20:11:10.931066 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:10.931032    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz"]
Sep 05 20:11:10.934745 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:10.934722    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz"
Sep 05 20:11:10.942047 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:10.942020    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz"]
Sep 05 20:11:11.070342 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:11.070305    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l7ft\" (UniqueName: \"kubernetes.io/projected/a86b2193-95ed-422f-9097-3bd6378b5e99-kube-api-access-5l7ft\") pod \"payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5\" (UID: \"a86b2193-95ed-422f-9097-3bd6378b5e99\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5"
Sep 05 20:11:11.070517 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:11.070417    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c614a8a-83e2-4e79-a921-83e3dd9afbe0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz\" (UID: \"9c614a8a-83e2-4e79-a921-83e3dd9afbe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz"
Sep 05 20:11:11.070517 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:11.070458    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pmgm\" (UniqueName: \"kubernetes.io/projected/9c614a8a-83e2-4e79-a921-83e3dd9afbe0-kube-api-access-5pmgm\") pod \"payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz\" (UID: \"9c614a8a-83e2-4e79-a921-83e3dd9afbe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz"
Sep 05 20:11:11.070613 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:11.070530    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a86b2193-95ed-422f-9097-3bd6378b5e99-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5\" (UID: \"a86b2193-95ed-422f-9097-3bd6378b5e99\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5"
Sep 05 20:11:11.171976 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:11.171884    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c614a8a-83e2-4e79-a921-83e3dd9afbe0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz\" (UID: \"9c614a8a-83e2-4e79-a921-83e3dd9afbe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz"
Sep 05 20:11:11.171976 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:11.171925    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pmgm\" (UniqueName: \"kubernetes.io/projected/9c614a8a-83e2-4e79-a921-83e3dd9afbe0-kube-api-access-5pmgm\") pod \"payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz\" (UID: \"9c614a8a-83e2-4e79-a921-83e3dd9afbe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz"
Sep 05 20:11:11.172225 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:11.172003    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a86b2193-95ed-422f-9097-3bd6378b5e99-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5\" (UID: \"a86b2193-95ed-422f-9097-3bd6378b5e99\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5"
Sep 05 20:11:11.172225 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:11.172048    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l7ft\" (UniqueName: \"kubernetes.io/projected/a86b2193-95ed-422f-9097-3bd6378b5e99-kube-api-access-5l7ft\") pod \"payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5\" (UID: \"a86b2193-95ed-422f-9097-3bd6378b5e99\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5"
Sep 05 20:11:11.172745 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:11.172723    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c614a8a-83e2-4e79-a921-83e3dd9afbe0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz\" (UID: \"9c614a8a-83e2-4e79-a921-83e3dd9afbe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz"
Sep 05 20:11:11.172930 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:11.172906    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a86b2193-95ed-422f-9097-3bd6378b5e99-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5\" (UID: \"a86b2193-95ed-422f-9097-3bd6378b5e99\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5"
Sep 05 20:11:11.181826 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:11.181798    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l7ft\" (UniqueName: \"kubernetes.io/projected/a86b2193-95ed-422f-9097-3bd6378b5e99-kube-api-access-5l7ft\") pod \"payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5\" (UID: \"a86b2193-95ed-422f-9097-3bd6378b5e99\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5"
Sep 05 20:11:11.181934 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:11.181844    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pmgm\" (UniqueName: \"kubernetes.io/projected/9c614a8a-83e2-4e79-a921-83e3dd9afbe0-kube-api-access-5pmgm\") pod \"payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz\" (UID: \"9c614a8a-83e2-4e79-a921-83e3dd9afbe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz"
Sep 05 20:11:11.217850 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:11.217818    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5"
Sep 05 20:11:11.247264 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:11.247232    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz"
Sep 05 20:11:11.493979 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:11.493952    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5"]
Sep 05 20:11:11.495046 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:11:11.495020    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda86b2193_95ed_422f_9097_3bd6378b5e99.slice/crio-afe0e846f1e7fbc5360d475f04cc23ceaafbcfa468baaf34829c2728e6ca4f1e WatchSource:0}: Error finding container afe0e846f1e7fbc5360d475f04cc23ceaafbcfa468baaf34829c2728e6ca4f1e: Status 404 returned error can't find the container with id afe0e846f1e7fbc5360d475f04cc23ceaafbcfa468baaf34829c2728e6ca4f1e
Sep 05 20:11:11.536427 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:11.536400    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz"]
Sep 05 20:11:11.537859 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:11:11.537829    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c614a8a_83e2_4e79_a921_83e3dd9afbe0.slice/crio-47228154a58ab8f7eba2f9743768c9a432a7f158f8f9ab6b27146de91ef0854d WatchSource:0}: Error finding container 47228154a58ab8f7eba2f9743768c9a432a7f158f8f9ab6b27146de91ef0854d: Status 404 returned error can't find the container with id 47228154a58ab8f7eba2f9743768c9a432a7f158f8f9ab6b27146de91ef0854d
Sep 05 20:11:12.124210 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:12.124173    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5" event={"ID":"a86b2193-95ed-422f-9097-3bd6378b5e99","Type":"ContainerStarted","Data":"5f1fc36004778008759e29fb9eb46bb7af390f85fdb3438dce2e1f87c6ecfaff"}
Sep 05 20:11:12.124691 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:12.124235    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5" event={"ID":"a86b2193-95ed-422f-9097-3bd6378b5e99","Type":"ContainerStarted","Data":"afe0e846f1e7fbc5360d475f04cc23ceaafbcfa468baaf34829c2728e6ca4f1e"}
Sep 05 20:11:12.124691 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:12.124378    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5"
Sep 05 20:11:12.125765 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:12.125736    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz" event={"ID":"9c614a8a-83e2-4e79-a921-83e3dd9afbe0","Type":"ContainerStarted","Data":"9c56f506f495681d73d7dfa16ce643107523b69df06205ee5662b7e8da889e6d"}
Sep 05 20:11:12.125887 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:12.125781    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz" event={"ID":"9c614a8a-83e2-4e79-a921-83e3dd9afbe0","Type":"ContainerStarted","Data":"47228154a58ab8f7eba2f9743768c9a432a7f158f8f9ab6b27146de91ef0854d"}
Sep 05 20:11:12.125887 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:12.125834    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz"
Sep 05 20:11:12.141502 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:12.141455    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5" podStartSLOduration=2.1414242310000002 podStartE2EDuration="2.141424231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:11:12.140471412 +0000 UTC m=+1671.170103330" watchObservedRunningTime="2026-09-05 20:11:12.141424231 +0000 UTC m=+1671.171056136"
Sep 05 20:11:12.158731 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:12.158680    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz" podStartSLOduration=2.158663351 podStartE2EDuration="2.158663351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:11:12.156699496 +0000 UTC m=+1671.186331402" watchObservedRunningTime="2026-09-05 20:11:12.158663351 +0000 UTC m=+1671.188295256"
Sep 05 20:11:23.134830 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:23.134796    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5"
Sep 05 20:11:23.135298 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:23.134864    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz"
Sep 05 20:11:39.117098 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:11:39.117067    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz_9c614a8a-83e2-4e79-a921-83e3dd9afbe0/payload-processing/0.log"
Sep 05 20:12:14.658619 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:14.658584    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz_99c1d1ac-b6c8-4047-83e8-168b8b94fc56/payload-processing/0.log"
Sep 05 20:12:14.890469 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:14.890439    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz_9c614a8a-83e2-4e79-a921-83e3dd9afbe0/payload-processing/0.log"
Sep 05 20:12:44.053024 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:44.052983    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-14a23e-2jpls"]
Sep 05 20:12:44.056524 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:44.056506    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-14a23e-2jpls"
Sep 05 20:12:44.058606 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:44.058589    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 05 20:12:44.058676 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:44.058589    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zx89\""
Sep 05 20:12:44.063579 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:44.063548    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-14a23e-2jpls"]
Sep 05 20:12:44.174597 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:44.174558    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67dc0206-4c02-45ff-85c2-26896c0102b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-14a23e-2jpls\" (UID: \"67dc0206-4c02-45ff-85c2-26896c0102b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-14a23e-2jpls"
Sep 05 20:12:44.275748 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:44.275711    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67dc0206-4c02-45ff-85c2-26896c0102b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-14a23e-2jpls\" (UID: \"67dc0206-4c02-45ff-85c2-26896c0102b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-14a23e-2jpls"
Sep 05 20:12:44.276406 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:44.276384    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67dc0206-4c02-45ff-85c2-26896c0102b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-14a23e-2jpls\" (UID: \"67dc0206-4c02-45ff-85c2-26896c0102b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-14a23e-2jpls"
Sep 05 20:12:44.370713 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:44.370675    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-14a23e-2jpls"
Sep 05 20:12:44.643123 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:44.643077    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-14a23e-2jpls"]
Sep 05 20:12:44.644867 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:12:44.644834    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67dc0206_4c02_45ff_85c2_26896c0102b1.slice/crio-3d09e556f6846f659f879b101f6fc443436712b1324f30e50c7c473c1e2385c3 WatchSource:0}: Error finding container 3d09e556f6846f659f879b101f6fc443436712b1324f30e50c7c473c1e2385c3: Status 404 returned error can't find the container with id 3d09e556f6846f659f879b101f6fc443436712b1324f30e50c7c473c1e2385c3
Sep 05 20:12:44.723650 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:44.723611    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-14a23e-2jpls" event={"ID":"67dc0206-4c02-45ff-85c2-26896c0102b1","Type":"ContainerStarted","Data":"3d09e556f6846f659f879b101f6fc443436712b1324f30e50c7c473c1e2385c3"}
Sep 05 20:12:45.732073 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:45.731982    2583 generic.go:356] "Generic (PLEG): container finished" podID="67dc0206-4c02-45ff-85c2-26896c0102b1" containerID="5a4aaa67cb7bc9097c37e94a2c227ef77b805fe8819cc7cc9d1c25f935c7b46d" exitCode=0
Sep 05 20:12:45.732449 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:45.732076    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-14a23e-2jpls" event={"ID":"67dc0206-4c02-45ff-85c2-26896c0102b1","Type":"ContainerDied","Data":"5a4aaa67cb7bc9097c37e94a2c227ef77b805fe8819cc7cc9d1c25f935c7b46d"}
Sep 05 20:12:46.887581 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:46.887549    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-14a23e-2jpls"
Sep 05 20:12:47.003645 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:47.003553    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67dc0206-4c02-45ff-85c2-26896c0102b1-maas-api-service-ca\") pod \"67dc0206-4c02-45ff-85c2-26896c0102b1\" (UID: \"67dc0206-4c02-45ff-85c2-26896c0102b1\") "
Sep 05 20:12:47.003990 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:47.003964    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67dc0206-4c02-45ff-85c2-26896c0102b1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "67dc0206-4c02-45ff-85c2-26896c0102b1" (UID: "67dc0206-4c02-45ff-85c2-26896c0102b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:12:47.104590 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:47.104549    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67dc0206-4c02-45ff-85c2-26896c0102b1-maas-api-service-ca\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:12:47.745855 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:47.745772    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-14a23e-2jpls"
Sep 05 20:12:47.745855 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:47.745809    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-14a23e-2jpls" event={"ID":"67dc0206-4c02-45ff-85c2-26896c0102b1","Type":"ContainerDied","Data":"3d09e556f6846f659f879b101f6fc443436712b1324f30e50c7c473c1e2385c3"}
Sep 05 20:12:47.745855 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:47.745856    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d09e556f6846f659f879b101f6fc443436712b1324f30e50c7c473c1e2385c3"
Sep 05 20:12:50.302797 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:50.302759    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-14a23e-2jpls"]
Sep 05 20:12:50.307238 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:50.307213    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-14a23e-2jpls"]
Sep 05 20:12:51.443162 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:12:51.443096    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67dc0206-4c02-45ff-85c2-26896c0102b1" path="/var/lib/kubelet/pods/67dc0206-4c02-45ff-85c2-26896c0102b1/volumes"
Sep 05 20:13:15.808725 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:15.808635    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz"]
Sep 05 20:13:15.809341 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:15.808906    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz" podUID="99c1d1ac-b6c8-4047-83e8-168b8b94fc56" containerName="payload-processing" containerID="cri-o://584de74700831f0c9965770e22182644db6b52f44f4177c39f41e24825ce5c8a" gracePeriod=30
Sep 05 20:13:15.817313 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:15.817276    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd"]
Sep 05 20:13:15.817560 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:15.817509    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd" podUID="067dea99-d8a2-4a9e-a804-f24b6d5421e2" containerName="payload-pre-processing" containerID="cri-o://a5d85a13c75e7a81d2144db48e1a2561b38256336e73f4613a6e637fa6fa98a2" gracePeriod=30
Sep 05 20:13:15.927677 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:15.927637    2583 generic.go:356] "Generic (PLEG): container finished" podID="99c1d1ac-b6c8-4047-83e8-168b8b94fc56" containerID="584de74700831f0c9965770e22182644db6b52f44f4177c39f41e24825ce5c8a" exitCode=0
Sep 05 20:13:15.927816 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:15.927706    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz" event={"ID":"99c1d1ac-b6c8-4047-83e8-168b8b94fc56","Type":"ContainerDied","Data":"584de74700831f0c9965770e22182644db6b52f44f4177c39f41e24825ce5c8a"}
Sep 05 20:13:16.104957 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.104923    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz"
Sep 05 20:13:16.107661 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.107641    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd"
Sep 05 20:13:16.186895 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.186857    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067dea99-d8a2-4a9e-a804-f24b6d5421e2-plugins-config-volume\") pod \"067dea99-d8a2-4a9e-a804-f24b6d5421e2\" (UID: \"067dea99-d8a2-4a9e-a804-f24b6d5421e2\") "
Sep 05 20:13:16.186895 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.186894    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99c1d1ac-b6c8-4047-83e8-168b8b94fc56-plugins-config-volume\") pod \"99c1d1ac-b6c8-4047-83e8-168b8b94fc56\" (UID: \"99c1d1ac-b6c8-4047-83e8-168b8b94fc56\") "
Sep 05 20:13:16.187159 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.186955    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2zqw\" (UniqueName: \"kubernetes.io/projected/99c1d1ac-b6c8-4047-83e8-168b8b94fc56-kube-api-access-s2zqw\") pod \"99c1d1ac-b6c8-4047-83e8-168b8b94fc56\" (UID: \"99c1d1ac-b6c8-4047-83e8-168b8b94fc56\") "
Sep 05 20:13:16.187159 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.186991    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z5t5\" (UniqueName: \"kubernetes.io/projected/067dea99-d8a2-4a9e-a804-f24b6d5421e2-kube-api-access-8z5t5\") pod \"067dea99-d8a2-4a9e-a804-f24b6d5421e2\" (UID: \"067dea99-d8a2-4a9e-a804-f24b6d5421e2\") "
Sep 05 20:13:16.187347 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.187322    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c1d1ac-b6c8-4047-83e8-168b8b94fc56-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "99c1d1ac-b6c8-4047-83e8-168b8b94fc56" (UID: "99c1d1ac-b6c8-4047-83e8-168b8b94fc56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:13:16.187393 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.187332    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067dea99-d8a2-4a9e-a804-f24b6d5421e2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "067dea99-d8a2-4a9e-a804-f24b6d5421e2" (UID: "067dea99-d8a2-4a9e-a804-f24b6d5421e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:13:16.189160 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.189131    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/067dea99-d8a2-4a9e-a804-f24b6d5421e2-kube-api-access-8z5t5" (OuterVolumeSpecName: "kube-api-access-8z5t5") pod "067dea99-d8a2-4a9e-a804-f24b6d5421e2" (UID: "067dea99-d8a2-4a9e-a804-f24b6d5421e2"). InnerVolumeSpecName "kube-api-access-8z5t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:13:16.189284 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.189185    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c1d1ac-b6c8-4047-83e8-168b8b94fc56-kube-api-access-s2zqw" (OuterVolumeSpecName: "kube-api-access-s2zqw") pod "99c1d1ac-b6c8-4047-83e8-168b8b94fc56" (UID: "99c1d1ac-b6c8-4047-83e8-168b8b94fc56"). InnerVolumeSpecName "kube-api-access-s2zqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:13:16.287849 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.287807    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2zqw\" (UniqueName: \"kubernetes.io/projected/99c1d1ac-b6c8-4047-83e8-168b8b94fc56-kube-api-access-s2zqw\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:13:16.287849 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.287841    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z5t5\" (UniqueName: \"kubernetes.io/projected/067dea99-d8a2-4a9e-a804-f24b6d5421e2-kube-api-access-8z5t5\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:13:16.287849 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.287852    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/067dea99-d8a2-4a9e-a804-f24b6d5421e2-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:13:16.288075 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.287861    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99c1d1ac-b6c8-4047-83e8-168b8b94fc56-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:13:16.935227 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.935193    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz"
Sep 05 20:13:16.935678 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.935191    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz" event={"ID":"99c1d1ac-b6c8-4047-83e8-168b8b94fc56","Type":"ContainerDied","Data":"310de22a5d47eb63e595eded2ae89393675bf399b7636eba2b3a47f24356772f"}
Sep 05 20:13:16.935678 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.935400    2583 scope.go:117] "RemoveContainer" containerID="584de74700831f0c9965770e22182644db6b52f44f4177c39f41e24825ce5c8a"
Sep 05 20:13:16.936793 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.936756    2583 generic.go:356] "Generic (PLEG): container finished" podID="067dea99-d8a2-4a9e-a804-f24b6d5421e2" containerID="a5d85a13c75e7a81d2144db48e1a2561b38256336e73f4613a6e637fa6fa98a2" exitCode=0
Sep 05 20:13:16.936793 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.936785    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd" event={"ID":"067dea99-d8a2-4a9e-a804-f24b6d5421e2","Type":"ContainerDied","Data":"a5d85a13c75e7a81d2144db48e1a2561b38256336e73f4613a6e637fa6fa98a2"}
Sep 05 20:13:16.936901 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.936829    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd" event={"ID":"067dea99-d8a2-4a9e-a804-f24b6d5421e2","Type":"ContainerDied","Data":"dcd07d94c820dac2bab1e69e38bad516a58d5a671082b4970f62d92c78e522cd"}
Sep 05 20:13:16.936901 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.936840    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd"
Sep 05 20:13:16.945829 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.945638    2583 scope.go:117] "RemoveContainer" containerID="a5d85a13c75e7a81d2144db48e1a2561b38256336e73f4613a6e637fa6fa98a2"
Sep 05 20:13:16.958738 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.958718    2583 scope.go:117] "RemoveContainer" containerID="a5d85a13c75e7a81d2144db48e1a2561b38256336e73f4613a6e637fa6fa98a2"
Sep 05 20:13:16.959085 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:13:16.959062    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5d85a13c75e7a81d2144db48e1a2561b38256336e73f4613a6e637fa6fa98a2\": container with ID starting with a5d85a13c75e7a81d2144db48e1a2561b38256336e73f4613a6e637fa6fa98a2 not found: ID does not exist" containerID="a5d85a13c75e7a81d2144db48e1a2561b38256336e73f4613a6e637fa6fa98a2"
Sep 05 20:13:16.959159 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.959098    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5d85a13c75e7a81d2144db48e1a2561b38256336e73f4613a6e637fa6fa98a2"} err="failed to get container status \"a5d85a13c75e7a81d2144db48e1a2561b38256336e73f4613a6e637fa6fa98a2\": rpc error: code = NotFound desc = could not find container \"a5d85a13c75e7a81d2144db48e1a2561b38256336e73f4613a6e637fa6fa98a2\": container with ID starting with a5d85a13c75e7a81d2144db48e1a2561b38256336e73f4613a6e637fa6fa98a2 not found: ID does not exist"
Sep 05 20:13:16.963206 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.963181    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd"]
Sep 05 20:13:16.969115 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.969090    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d0f6b5-f8cbf4fc4-7r2rd"]
Sep 05 20:13:16.977867 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.977836    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz"]
Sep 05 20:13:16.981672 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:16.981648    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d0f6b5-5cb79b5679-kw2bz"]
Sep 05 20:13:17.435775 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:17.435743    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="067dea99-d8a2-4a9e-a804-f24b6d5421e2" path="/var/lib/kubelet/pods/067dea99-d8a2-4a9e-a804-f24b6d5421e2/volumes"
Sep 05 20:13:17.436118 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:17.436098    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99c1d1ac-b6c8-4047-83e8-168b8b94fc56" path="/var/lib/kubelet/pods/99c1d1ac-b6c8-4047-83e8-168b8b94fc56/volumes"
Sep 05 20:13:21.604059 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:21.604020    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf9tj_f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b/ovn-acl-logging/0.log"
Sep 05 20:13:21.613711 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:21.613678    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf9tj_f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b/ovn-acl-logging/0.log"
Sep 05 20:13:21.916865 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:13:21.916780    2583 scope.go:117] "RemoveContainer" containerID="b0ad4fa5e4967340b4da4c45906e37c489c7a14c410943e4ccd327eadff1ea88"
Sep 05 20:14:15.760839 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:15.760794    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz"]
Sep 05 20:14:15.761446 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:15.761010    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz" podUID="9c614a8a-83e2-4e79-a921-83e3dd9afbe0" containerName="payload-processing" containerID="cri-o://9c56f506f495681d73d7dfa16ce643107523b69df06205ee5662b7e8da889e6d" gracePeriod=30
Sep 05 20:14:15.768661 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:15.768297    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5"]
Sep 05 20:14:15.768750 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:15.768648    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5" podUID="a86b2193-95ed-422f-9097-3bd6378b5e99" containerName="payload-pre-processing" containerID="cri-o://5f1fc36004778008759e29fb9eb46bb7af390f85fdb3438dce2e1f87c6ecfaff" gracePeriod=30
Sep 05 20:14:16.075495 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.075157    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz"
Sep 05 20:14:16.080502 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.080475    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5"
Sep 05 20:14:16.153128 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.153074    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5l7ft\" (UniqueName: \"kubernetes.io/projected/a86b2193-95ed-422f-9097-3bd6378b5e99-kube-api-access-5l7ft\") pod \"a86b2193-95ed-422f-9097-3bd6378b5e99\" (UID: \"a86b2193-95ed-422f-9097-3bd6378b5e99\") "
Sep 05 20:14:16.153278 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.153158    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c614a8a-83e2-4e79-a921-83e3dd9afbe0-plugins-config-volume\") pod \"9c614a8a-83e2-4e79-a921-83e3dd9afbe0\" (UID: \"9c614a8a-83e2-4e79-a921-83e3dd9afbe0\") "
Sep 05 20:14:16.153278 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.153199    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a86b2193-95ed-422f-9097-3bd6378b5e99-plugins-config-volume\") pod \"a86b2193-95ed-422f-9097-3bd6378b5e99\" (UID: \"a86b2193-95ed-422f-9097-3bd6378b5e99\") "
Sep 05 20:14:16.153278 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.153223    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pmgm\" (UniqueName: \"kubernetes.io/projected/9c614a8a-83e2-4e79-a921-83e3dd9afbe0-kube-api-access-5pmgm\") pod \"9c614a8a-83e2-4e79-a921-83e3dd9afbe0\" (UID: \"9c614a8a-83e2-4e79-a921-83e3dd9afbe0\") "
Sep 05 20:14:16.153513 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.153490    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c614a8a-83e2-4e79-a921-83e3dd9afbe0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9c614a8a-83e2-4e79-a921-83e3dd9afbe0" (UID: "9c614a8a-83e2-4e79-a921-83e3dd9afbe0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:14:16.153616 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.153593    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a86b2193-95ed-422f-9097-3bd6378b5e99-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a86b2193-95ed-422f-9097-3bd6378b5e99" (UID: "a86b2193-95ed-422f-9097-3bd6378b5e99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:14:16.155379 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.155357    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a86b2193-95ed-422f-9097-3bd6378b5e99-kube-api-access-5l7ft" (OuterVolumeSpecName: "kube-api-access-5l7ft") pod "a86b2193-95ed-422f-9097-3bd6378b5e99" (UID: "a86b2193-95ed-422f-9097-3bd6378b5e99"). InnerVolumeSpecName "kube-api-access-5l7ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:14:16.155446 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.155422    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c614a8a-83e2-4e79-a921-83e3dd9afbe0-kube-api-access-5pmgm" (OuterVolumeSpecName: "kube-api-access-5pmgm") pod "9c614a8a-83e2-4e79-a921-83e3dd9afbe0" (UID: "9c614a8a-83e2-4e79-a921-83e3dd9afbe0"). InnerVolumeSpecName "kube-api-access-5pmgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:14:16.254545 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.254498    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l7ft\" (UniqueName: \"kubernetes.io/projected/a86b2193-95ed-422f-9097-3bd6378b5e99-kube-api-access-5l7ft\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:14:16.254545 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.254535    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c614a8a-83e2-4e79-a921-83e3dd9afbe0-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:14:16.254545 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.254548    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a86b2193-95ed-422f-9097-3bd6378b5e99-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:14:16.254793 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.254561    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pmgm\" (UniqueName: \"kubernetes.io/projected/9c614a8a-83e2-4e79-a921-83e3dd9afbe0-kube-api-access-5pmgm\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:14:16.307086 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.306994    2583 generic.go:356] "Generic (PLEG): container finished" podID="a86b2193-95ed-422f-9097-3bd6378b5e99" containerID="5f1fc36004778008759e29fb9eb46bb7af390f85fdb3438dce2e1f87c6ecfaff" exitCode=0
Sep 05 20:14:16.307086 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.307042    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5"
Sep 05 20:14:16.307345 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.307081    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5" event={"ID":"a86b2193-95ed-422f-9097-3bd6378b5e99","Type":"ContainerDied","Data":"5f1fc36004778008759e29fb9eb46bb7af390f85fdb3438dce2e1f87c6ecfaff"}
Sep 05 20:14:16.307345 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.307160    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5" event={"ID":"a86b2193-95ed-422f-9097-3bd6378b5e99","Type":"ContainerDied","Data":"afe0e846f1e7fbc5360d475f04cc23ceaafbcfa468baaf34829c2728e6ca4f1e"}
Sep 05 20:14:16.307345 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.307185    2583 scope.go:117] "RemoveContainer" containerID="5f1fc36004778008759e29fb9eb46bb7af390f85fdb3438dce2e1f87c6ecfaff"
Sep 05 20:14:16.308727 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.308706    2583 generic.go:356] "Generic (PLEG): container finished" podID="9c614a8a-83e2-4e79-a921-83e3dd9afbe0" containerID="9c56f506f495681d73d7dfa16ce643107523b69df06205ee5662b7e8da889e6d" exitCode=0
Sep 05 20:14:16.308845 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.308751    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz"
Sep 05 20:14:16.308845 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.308752    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz" event={"ID":"9c614a8a-83e2-4e79-a921-83e3dd9afbe0","Type":"ContainerDied","Data":"9c56f506f495681d73d7dfa16ce643107523b69df06205ee5662b7e8da889e6d"}
Sep 05 20:14:16.308975 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.308850    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz" event={"ID":"9c614a8a-83e2-4e79-a921-83e3dd9afbe0","Type":"ContainerDied","Data":"47228154a58ab8f7eba2f9743768c9a432a7f158f8f9ab6b27146de91ef0854d"}
Sep 05 20:14:16.318485 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.318467    2583 scope.go:117] "RemoveContainer" containerID="5f1fc36004778008759e29fb9eb46bb7af390f85fdb3438dce2e1f87c6ecfaff"
Sep 05 20:14:16.318809 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:14:16.318783    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f1fc36004778008759e29fb9eb46bb7af390f85fdb3438dce2e1f87c6ecfaff\": container with ID starting with 5f1fc36004778008759e29fb9eb46bb7af390f85fdb3438dce2e1f87c6ecfaff not found: ID does not exist" containerID="5f1fc36004778008759e29fb9eb46bb7af390f85fdb3438dce2e1f87c6ecfaff"
Sep 05 20:14:16.318919 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.318819    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f1fc36004778008759e29fb9eb46bb7af390f85fdb3438dce2e1f87c6ecfaff"} err="failed to get container status \"5f1fc36004778008759e29fb9eb46bb7af390f85fdb3438dce2e1f87c6ecfaff\": rpc error: code = NotFound desc = could not find container \"5f1fc36004778008759e29fb9eb46bb7af390f85fdb3438dce2e1f87c6ecfaff\": container with ID starting with 5f1fc36004778008759e29fb9eb46bb7af390f85fdb3438dce2e1f87c6ecfaff not found: ID does not exist"
Sep 05 20:14:16.318919 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.318841    2583 scope.go:117] "RemoveContainer" containerID="9c56f506f495681d73d7dfa16ce643107523b69df06205ee5662b7e8da889e6d"
Sep 05 20:14:16.328164 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.328138    2583 scope.go:117] "RemoveContainer" containerID="9c56f506f495681d73d7dfa16ce643107523b69df06205ee5662b7e8da889e6d"
Sep 05 20:14:16.328841 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:14:16.328562    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c56f506f495681d73d7dfa16ce643107523b69df06205ee5662b7e8da889e6d\": container with ID starting with 9c56f506f495681d73d7dfa16ce643107523b69df06205ee5662b7e8da889e6d not found: ID does not exist" containerID="9c56f506f495681d73d7dfa16ce643107523b69df06205ee5662b7e8da889e6d"
Sep 05 20:14:16.328841 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.328624    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c56f506f495681d73d7dfa16ce643107523b69df06205ee5662b7e8da889e6d"} err="failed to get container status \"9c56f506f495681d73d7dfa16ce643107523b69df06205ee5662b7e8da889e6d\": rpc error: code = NotFound desc = could not find container \"9c56f506f495681d73d7dfa16ce643107523b69df06205ee5662b7e8da889e6d\": container with ID starting with 9c56f506f495681d73d7dfa16ce643107523b69df06205ee5662b7e8da889e6d not found: ID does not exist"
Sep 05 20:14:16.330139 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.330094    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5"]
Sep 05 20:14:16.332598 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.332578    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6693ae-7cff786886-b7cl5"]
Sep 05 20:14:16.341368 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.341347    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz"]
Sep 05 20:14:16.345536 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:16.345513    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6693ae-6f54775b67-jrsnz"]
Sep 05 20:14:17.434962 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:17.434929    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c614a8a-83e2-4e79-a921-83e3dd9afbe0" path="/var/lib/kubelet/pods/9c614a8a-83e2-4e79-a921-83e3dd9afbe0/volumes"
Sep 05 20:14:17.435359 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:17.435326    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a86b2193-95ed-422f-9097-3bd6378b5e99" path="/var/lib/kubelet/pods/a86b2193-95ed-422f-9097-3bd6378b5e99/volumes"
Sep 05 20:14:33.882747 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:33.882350    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl"]
Sep 05 20:14:33.882747 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:33.882620    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl" podUID="be0ebded-91d9-40b0-87cd-52a3a9093edd" containerName="payload-processing" containerID="cri-o://fc0e7064fc100cc6da4913c21516481d986f983c51fcb7dedb7dc4043d66fcf0" gracePeriod=30
Sep 05 20:14:33.887895 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:33.887865    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf"]
Sep 05 20:14:33.888501 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:33.888451    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf" podUID="e4cfd886-4a10-4b97-b3bf-e7bd86e8b740" containerName="payload-pre-processing" containerID="cri-o://1fdbb299ee8d2887b29166223d5839815c94ecfd8a7e9deb29fa2a0653fdab90" gracePeriod=30
Sep 05 20:14:34.153687 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.153662    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl"
Sep 05 20:14:34.186540 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.186508    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf"
Sep 05 20:14:34.226639 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.226604    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcx8k\" (UniqueName: \"kubernetes.io/projected/be0ebded-91d9-40b0-87cd-52a3a9093edd-kube-api-access-wcx8k\") pod \"be0ebded-91d9-40b0-87cd-52a3a9093edd\" (UID: \"be0ebded-91d9-40b0-87cd-52a3a9093edd\") "
Sep 05 20:14:34.226791 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.226661    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be0ebded-91d9-40b0-87cd-52a3a9093edd-plugins-config-volume\") pod \"be0ebded-91d9-40b0-87cd-52a3a9093edd\" (UID: \"be0ebded-91d9-40b0-87cd-52a3a9093edd\") "
Sep 05 20:14:34.227045 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.227015    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be0ebded-91d9-40b0-87cd-52a3a9093edd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "be0ebded-91d9-40b0-87cd-52a3a9093edd" (UID: "be0ebded-91d9-40b0-87cd-52a3a9093edd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:14:34.228811 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.228787    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be0ebded-91d9-40b0-87cd-52a3a9093edd-kube-api-access-wcx8k" (OuterVolumeSpecName: "kube-api-access-wcx8k") pod "be0ebded-91d9-40b0-87cd-52a3a9093edd" (UID: "be0ebded-91d9-40b0-87cd-52a3a9093edd"). InnerVolumeSpecName "kube-api-access-wcx8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:14:34.327814 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.327779    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4cfd886-4a10-4b97-b3bf-e7bd86e8b740-plugins-config-volume\") pod \"e4cfd886-4a10-4b97-b3bf-e7bd86e8b740\" (UID: \"e4cfd886-4a10-4b97-b3bf-e7bd86e8b740\") "
Sep 05 20:14:34.327991 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.327829    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qp9fg\" (UniqueName: \"kubernetes.io/projected/e4cfd886-4a10-4b97-b3bf-e7bd86e8b740-kube-api-access-qp9fg\") pod \"e4cfd886-4a10-4b97-b3bf-e7bd86e8b740\" (UID: \"e4cfd886-4a10-4b97-b3bf-e7bd86e8b740\") "
Sep 05 20:14:34.328036 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.328008    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcx8k\" (UniqueName: \"kubernetes.io/projected/be0ebded-91d9-40b0-87cd-52a3a9093edd-kube-api-access-wcx8k\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:14:34.328036 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.328018    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be0ebded-91d9-40b0-87cd-52a3a9093edd-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:14:34.328244 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.328216    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4cfd886-4a10-4b97-b3bf-e7bd86e8b740-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4cfd886-4a10-4b97-b3bf-e7bd86e8b740" (UID: "e4cfd886-4a10-4b97-b3bf-e7bd86e8b740"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:14:34.329958 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.329935    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4cfd886-4a10-4b97-b3bf-e7bd86e8b740-kube-api-access-qp9fg" (OuterVolumeSpecName: "kube-api-access-qp9fg") pod "e4cfd886-4a10-4b97-b3bf-e7bd86e8b740" (UID: "e4cfd886-4a10-4b97-b3bf-e7bd86e8b740"). InnerVolumeSpecName "kube-api-access-qp9fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:14:34.428756 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.428663    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4cfd886-4a10-4b97-b3bf-e7bd86e8b740-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:14:34.428756 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.428701    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp9fg\" (UniqueName: \"kubernetes.io/projected/e4cfd886-4a10-4b97-b3bf-e7bd86e8b740-kube-api-access-qp9fg\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:14:34.432295 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.432269    2583 generic.go:356] "Generic (PLEG): container finished" podID="be0ebded-91d9-40b0-87cd-52a3a9093edd" containerID="fc0e7064fc100cc6da4913c21516481d986f983c51fcb7dedb7dc4043d66fcf0" exitCode=0
Sep 05 20:14:34.432430 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.432299    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl"
Sep 05 20:14:34.432430 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.432347    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl" event={"ID":"be0ebded-91d9-40b0-87cd-52a3a9093edd","Type":"ContainerDied","Data":"fc0e7064fc100cc6da4913c21516481d986f983c51fcb7dedb7dc4043d66fcf0"}
Sep 05 20:14:34.432430 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.432395    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl" event={"ID":"be0ebded-91d9-40b0-87cd-52a3a9093edd","Type":"ContainerDied","Data":"734a64b59bb2dd64434b49af4da5a670dddb116ed77de121cf0eac0877708079"}
Sep 05 20:14:34.432430 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.432424    2583 scope.go:117] "RemoveContainer" containerID="fc0e7064fc100cc6da4913c21516481d986f983c51fcb7dedb7dc4043d66fcf0"
Sep 05 20:14:34.433756 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.433728    2583 generic.go:356] "Generic (PLEG): container finished" podID="e4cfd886-4a10-4b97-b3bf-e7bd86e8b740" containerID="1fdbb299ee8d2887b29166223d5839815c94ecfd8a7e9deb29fa2a0653fdab90" exitCode=0
Sep 05 20:14:34.433842 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.433764    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf"
Sep 05 20:14:34.433842 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.433810    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf" event={"ID":"e4cfd886-4a10-4b97-b3bf-e7bd86e8b740","Type":"ContainerDied","Data":"1fdbb299ee8d2887b29166223d5839815c94ecfd8a7e9deb29fa2a0653fdab90"}
Sep 05 20:14:34.433954 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.433860    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf" event={"ID":"e4cfd886-4a10-4b97-b3bf-e7bd86e8b740","Type":"ContainerDied","Data":"44fa6c0814b82911c40ab50d7e76f85989f5eb2e7b8fb7342793b6172d4e4911"}
Sep 05 20:14:34.442486 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.442440    2583 scope.go:117] "RemoveContainer" containerID="fc0e7064fc100cc6da4913c21516481d986f983c51fcb7dedb7dc4043d66fcf0"
Sep 05 20:14:34.442767 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:14:34.442744    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc0e7064fc100cc6da4913c21516481d986f983c51fcb7dedb7dc4043d66fcf0\": container with ID starting with fc0e7064fc100cc6da4913c21516481d986f983c51fcb7dedb7dc4043d66fcf0 not found: ID does not exist" containerID="fc0e7064fc100cc6da4913c21516481d986f983c51fcb7dedb7dc4043d66fcf0"
Sep 05 20:14:34.442861 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.442773    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc0e7064fc100cc6da4913c21516481d986f983c51fcb7dedb7dc4043d66fcf0"} err="failed to get container status \"fc0e7064fc100cc6da4913c21516481d986f983c51fcb7dedb7dc4043d66fcf0\": rpc error: code = NotFound desc = could not find container \"fc0e7064fc100cc6da4913c21516481d986f983c51fcb7dedb7dc4043d66fcf0\": container with ID starting with fc0e7064fc100cc6da4913c21516481d986f983c51fcb7dedb7dc4043d66fcf0 not found: ID does not exist"
Sep 05 20:14:34.442861 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.442788    2583 scope.go:117] "RemoveContainer" containerID="1fdbb299ee8d2887b29166223d5839815c94ecfd8a7e9deb29fa2a0653fdab90"
Sep 05 20:14:34.451278 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.451262    2583 scope.go:117] "RemoveContainer" containerID="1fdbb299ee8d2887b29166223d5839815c94ecfd8a7e9deb29fa2a0653fdab90"
Sep 05 20:14:34.451534 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:14:34.451514    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fdbb299ee8d2887b29166223d5839815c94ecfd8a7e9deb29fa2a0653fdab90\": container with ID starting with 1fdbb299ee8d2887b29166223d5839815c94ecfd8a7e9deb29fa2a0653fdab90 not found: ID does not exist" containerID="1fdbb299ee8d2887b29166223d5839815c94ecfd8a7e9deb29fa2a0653fdab90"
Sep 05 20:14:34.451597 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.451539    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fdbb299ee8d2887b29166223d5839815c94ecfd8a7e9deb29fa2a0653fdab90"} err="failed to get container status \"1fdbb299ee8d2887b29166223d5839815c94ecfd8a7e9deb29fa2a0653fdab90\": rpc error: code = NotFound desc = could not find container \"1fdbb299ee8d2887b29166223d5839815c94ecfd8a7e9deb29fa2a0653fdab90\": container with ID starting with 1fdbb299ee8d2887b29166223d5839815c94ecfd8a7e9deb29fa2a0653fdab90 not found: ID does not exist"
Sep 05 20:14:34.461750 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.461728    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl"]
Sep 05 20:14:34.465881 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.465859    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-4d1885-54769bb8dc-kcsnl"]
Sep 05 20:14:34.476387 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.476355    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf"]
Sep 05 20:14:34.480723 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:34.480701    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-4d1885-6f6c55f958-zwdrf"]
Sep 05 20:14:35.435288 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:35.435243    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be0ebded-91d9-40b0-87cd-52a3a9093edd" path="/var/lib/kubelet/pods/be0ebded-91d9-40b0-87cd-52a3a9093edd/volumes"
Sep 05 20:14:35.435868 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:35.435846    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4cfd886-4a10-4b97-b3bf-e7bd86e8b740" path="/var/lib/kubelet/pods/e4cfd886-4a10-4b97-b3bf-e7bd86e8b740/volumes"
Sep 05 20:14:40.875751 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:40.875712    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"]
Sep 05 20:14:40.879406 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:40.875994    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn" podUID="12890272-a09c-4655-a206-464c55195412" containerName="istio-proxy" containerID="cri-o://b1a854d1809007fa2ceff39deded1d1d41dd09e003c75c7ba32adc975d7ab811" gracePeriod=30
Sep 05 20:14:46.134231 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.134203    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:14:46.247668 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.247568    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-credential-socket\") pod \"12890272-a09c-4655-a206-464c55195412\" (UID: \"12890272-a09c-4655-a206-464c55195412\") "
Sep 05 20:14:46.247668 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.247627    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-workload-socket\") pod \"12890272-a09c-4655-a206-464c55195412\" (UID: \"12890272-a09c-4655-a206-464c55195412\") "
Sep 05 20:14:46.247917 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.247681    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-istio-data\") pod \"12890272-a09c-4655-a206-464c55195412\" (UID: \"12890272-a09c-4655-a206-464c55195412\") "
Sep 05 20:14:46.247917 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.247716    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-workload-certs\") pod \"12890272-a09c-4655-a206-464c55195412\" (UID: \"12890272-a09c-4655-a206-464c55195412\") "
Sep 05 20:14:46.247917 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.247763    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9l98l\" (UniqueName: \"kubernetes.io/projected/12890272-a09c-4655-a206-464c55195412-kube-api-access-9l98l\") pod \"12890272-a09c-4655-a206-464c55195412\" (UID: \"12890272-a09c-4655-a206-464c55195412\") "
Sep 05 20:14:46.247917 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.247784    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-istio-envoy\") pod \"12890272-a09c-4655-a206-464c55195412\" (UID: \"12890272-a09c-4655-a206-464c55195412\") "
Sep 05 20:14:46.247917 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.247806    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12890272-a09c-4655-a206-464c55195412-istiod-ca-cert\") pod \"12890272-a09c-4655-a206-464c55195412\" (UID: \"12890272-a09c-4655-a206-464c55195412\") "
Sep 05 20:14:46.247917 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.247839    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12890272-a09c-4655-a206-464c55195412-istio-token\") pod \"12890272-a09c-4655-a206-464c55195412\" (UID: \"12890272-a09c-4655-a206-464c55195412\") "
Sep 05 20:14:46.248262 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.247912    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "12890272-a09c-4655-a206-464c55195412" (UID: "12890272-a09c-4655-a206-464c55195412"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:14:46.248262 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.247968    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-istio-data" (OuterVolumeSpecName: "istio-data") pod "12890272-a09c-4655-a206-464c55195412" (UID: "12890272-a09c-4655-a206-464c55195412"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:14:46.248262 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.247995    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12890272-a09c-4655-a206-464c55195412-istio-podinfo\") pod \"12890272-a09c-4655-a206-464c55195412\" (UID: \"12890272-a09c-4655-a206-464c55195412\") "
Sep 05 20:14:46.248262 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.248037    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "12890272-a09c-4655-a206-464c55195412" (UID: "12890272-a09c-4655-a206-464c55195412"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:14:46.248447 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.248393    2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-credential-socket\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:14:46.248447 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.248421    2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-workload-socket\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:14:46.248447 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.248441    2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-istio-data\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:14:46.248554 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.248391    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12890272-a09c-4655-a206-464c55195412-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "12890272-a09c-4655-a206-464c55195412" (UID: "12890272-a09c-4655-a206-464c55195412"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:14:46.248554 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.248398    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "12890272-a09c-4655-a206-464c55195412" (UID: "12890272-a09c-4655-a206-464c55195412"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:14:46.250129 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.250094    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12890272-a09c-4655-a206-464c55195412-istio-token" (OuterVolumeSpecName: "istio-token") pod "12890272-a09c-4655-a206-464c55195412" (UID: "12890272-a09c-4655-a206-464c55195412"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:14:46.250223 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.250212    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12890272-a09c-4655-a206-464c55195412-kube-api-access-9l98l" (OuterVolumeSpecName: "kube-api-access-9l98l") pod "12890272-a09c-4655-a206-464c55195412" (UID: "12890272-a09c-4655-a206-464c55195412"). InnerVolumeSpecName "kube-api-access-9l98l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:14:46.250465 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.250451    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "12890272-a09c-4655-a206-464c55195412" (UID: "12890272-a09c-4655-a206-464c55195412"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:14:46.250580 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.250561    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12890272-a09c-4655-a206-464c55195412-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "12890272-a09c-4655-a206-464c55195412" (UID: "12890272-a09c-4655-a206-464c55195412"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 05 20:14:46.349335 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.349292    2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-workload-certs\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:14:46.349335 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.349328    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l98l\" (UniqueName: \"kubernetes.io/projected/12890272-a09c-4655-a206-464c55195412-kube-api-access-9l98l\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:14:46.349335 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.349342    2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12890272-a09c-4655-a206-464c55195412-istio-envoy\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:14:46.349584 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.349355    2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12890272-a09c-4655-a206-464c55195412-istiod-ca-cert\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:14:46.349584 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.349366    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12890272-a09c-4655-a206-464c55195412-istio-token\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:14:46.349584 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.349377    2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12890272-a09c-4655-a206-464c55195412-istio-podinfo\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:14:46.509358 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.509260    2583 generic.go:356] "Generic (PLEG): container finished" podID="12890272-a09c-4655-a206-464c55195412" containerID="b1a854d1809007fa2ceff39deded1d1d41dd09e003c75c7ba32adc975d7ab811" exitCode=0
Sep 05 20:14:46.509358 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.509310    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"
Sep 05 20:14:46.509358 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.509350    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn" event={"ID":"12890272-a09c-4655-a206-464c55195412","Type":"ContainerDied","Data":"b1a854d1809007fa2ceff39deded1d1d41dd09e003c75c7ba32adc975d7ab811"}
Sep 05 20:14:46.509632 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.509407    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn" event={"ID":"12890272-a09c-4655-a206-464c55195412","Type":"ContainerDied","Data":"d48e834bf01f60c027e3e95eddddc2aed084ca0e22ae5d7a310888f938873ab8"}
Sep 05 20:14:46.509632 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.509434    2583 scope.go:117] "RemoveContainer" containerID="b1a854d1809007fa2ceff39deded1d1d41dd09e003c75c7ba32adc975d7ab811"
Sep 05 20:14:46.519211 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.519191    2583 scope.go:117] "RemoveContainer" containerID="433e41c95469c3be61ac036ee1c2e97249f0376dec32ba4f60fe88792869a957"
Sep 05 20:14:46.527708 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.527684    2583 scope.go:117] "RemoveContainer" containerID="b1a854d1809007fa2ceff39deded1d1d41dd09e003c75c7ba32adc975d7ab811"
Sep 05 20:14:46.527988 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:14:46.527969    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a854d1809007fa2ceff39deded1d1d41dd09e003c75c7ba32adc975d7ab811\": container with ID starting with b1a854d1809007fa2ceff39deded1d1d41dd09e003c75c7ba32adc975d7ab811 not found: ID does not exist" containerID="b1a854d1809007fa2ceff39deded1d1d41dd09e003c75c7ba32adc975d7ab811"
Sep 05 20:14:46.528042 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.527998    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a854d1809007fa2ceff39deded1d1d41dd09e003c75c7ba32adc975d7ab811"} err="failed to get container status \"b1a854d1809007fa2ceff39deded1d1d41dd09e003c75c7ba32adc975d7ab811\": rpc error: code = NotFound desc = could not find container \"b1a854d1809007fa2ceff39deded1d1d41dd09e003c75c7ba32adc975d7ab811\": container with ID starting with b1a854d1809007fa2ceff39deded1d1d41dd09e003c75c7ba32adc975d7ab811 not found: ID does not exist"
Sep 05 20:14:46.528042 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.528016    2583 scope.go:117] "RemoveContainer" containerID="433e41c95469c3be61ac036ee1c2e97249f0376dec32ba4f60fe88792869a957"
Sep 05 20:14:46.528303 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:14:46.528287    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"433e41c95469c3be61ac036ee1c2e97249f0376dec32ba4f60fe88792869a957\": container with ID starting with 433e41c95469c3be61ac036ee1c2e97249f0376dec32ba4f60fe88792869a957 not found: ID does not exist" containerID="433e41c95469c3be61ac036ee1c2e97249f0376dec32ba4f60fe88792869a957"
Sep 05 20:14:46.528364 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.528307    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"433e41c95469c3be61ac036ee1c2e97249f0376dec32ba4f60fe88792869a957"} err="failed to get container status \"433e41c95469c3be61ac036ee1c2e97249f0376dec32ba4f60fe88792869a957\": rpc error: code = NotFound desc = could not find container \"433e41c95469c3be61ac036ee1c2e97249f0376dec32ba4f60fe88792869a957\": container with ID starting with 433e41c95469c3be61ac036ee1c2e97249f0376dec32ba4f60fe88792869a957 not found: ID does not exist"
Sep 05 20:14:46.532230 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.532205    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"]
Sep 05 20:14:46.536693 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:46.536672    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4d1885-openshift-default-6c8957b9f6-6fbcn"]
Sep 05 20:14:47.435366 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:47.435328    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12890272-a09c-4655-a206-464c55195412" path="/var/lib/kubelet/pods/12890272-a09c-4655-a206-464c55195412/volumes"
Sep 05 20:14:51.765942 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:51.765155    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k"]
Sep 05 20:14:51.765942 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:51.765419    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k" podUID="7272109d-9e39-4e5b-ba59-ff514f403aec" containerName="payload-processing" containerID="cri-o://9925c6014c39219963e56d6239a91f45b5709e593c7c56a633ebfaf2539be242" gracePeriod=30
Sep 05 20:14:51.773365 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:51.773338    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz"]
Sep 05 20:14:51.773764 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:51.773714    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz" podUID="dc3ec0e0-57b6-4289-910c-6cf10b36102b" containerName="payload-pre-processing" containerID="cri-o://4c3df131b361c31ad4482abaa95e02dbe354171c34ab951b24d28547edf75f2b" gracePeriod=30
Sep 05 20:14:52.054744 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.054717    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k"
Sep 05 20:14:52.057363 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.057342    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz"
Sep 05 20:14:52.207594 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.207562    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3ec0e0-57b6-4289-910c-6cf10b36102b-plugins-config-volume\") pod \"dc3ec0e0-57b6-4289-910c-6cf10b36102b\" (UID: \"dc3ec0e0-57b6-4289-910c-6cf10b36102b\") "
Sep 05 20:14:52.207761 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.207604    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7272109d-9e39-4e5b-ba59-ff514f403aec-plugins-config-volume\") pod \"7272109d-9e39-4e5b-ba59-ff514f403aec\" (UID: \"7272109d-9e39-4e5b-ba59-ff514f403aec\") "
Sep 05 20:14:52.207761 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.207637    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drhnk\" (UniqueName: \"kubernetes.io/projected/7272109d-9e39-4e5b-ba59-ff514f403aec-kube-api-access-drhnk\") pod \"7272109d-9e39-4e5b-ba59-ff514f403aec\" (UID: \"7272109d-9e39-4e5b-ba59-ff514f403aec\") "
Sep 05 20:14:52.207761 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.207701    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxdk9\" (UniqueName: \"kubernetes.io/projected/dc3ec0e0-57b6-4289-910c-6cf10b36102b-kube-api-access-jxdk9\") pod \"dc3ec0e0-57b6-4289-910c-6cf10b36102b\" (UID: \"dc3ec0e0-57b6-4289-910c-6cf10b36102b\") "
Sep 05 20:14:52.208182 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.208152    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7272109d-9e39-4e5b-ba59-ff514f403aec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7272109d-9e39-4e5b-ba59-ff514f403aec" (UID: "7272109d-9e39-4e5b-ba59-ff514f403aec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:14:52.208270 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.208171    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc3ec0e0-57b6-4289-910c-6cf10b36102b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dc3ec0e0-57b6-4289-910c-6cf10b36102b" (UID: "dc3ec0e0-57b6-4289-910c-6cf10b36102b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 05 20:14:52.209802 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.209778    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3ec0e0-57b6-4289-910c-6cf10b36102b-kube-api-access-jxdk9" (OuterVolumeSpecName: "kube-api-access-jxdk9") pod "dc3ec0e0-57b6-4289-910c-6cf10b36102b" (UID: "dc3ec0e0-57b6-4289-910c-6cf10b36102b"). InnerVolumeSpecName "kube-api-access-jxdk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:14:52.209960 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.209937    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7272109d-9e39-4e5b-ba59-ff514f403aec-kube-api-access-drhnk" (OuterVolumeSpecName: "kube-api-access-drhnk") pod "7272109d-9e39-4e5b-ba59-ff514f403aec" (UID: "7272109d-9e39-4e5b-ba59-ff514f403aec"). InnerVolumeSpecName "kube-api-access-drhnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:14:52.308896 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.308804    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7272109d-9e39-4e5b-ba59-ff514f403aec-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:14:52.308896 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.308838    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drhnk\" (UniqueName: \"kubernetes.io/projected/7272109d-9e39-4e5b-ba59-ff514f403aec-kube-api-access-drhnk\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:14:52.308896 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.308848    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxdk9\" (UniqueName: \"kubernetes.io/projected/dc3ec0e0-57b6-4289-910c-6cf10b36102b-kube-api-access-jxdk9\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:14:52.308896 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.308858    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3ec0e0-57b6-4289-910c-6cf10b36102b-plugins-config-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:14:52.548452 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.548411    2583 generic.go:356] "Generic (PLEG): container finished" podID="7272109d-9e39-4e5b-ba59-ff514f403aec" containerID="9925c6014c39219963e56d6239a91f45b5709e593c7c56a633ebfaf2539be242" exitCode=0
Sep 05 20:14:52.548655 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.548459    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k"
Sep 05 20:14:52.548655 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.548497    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k" event={"ID":"7272109d-9e39-4e5b-ba59-ff514f403aec","Type":"ContainerDied","Data":"9925c6014c39219963e56d6239a91f45b5709e593c7c56a633ebfaf2539be242"}
Sep 05 20:14:52.548655 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.548562    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k" event={"ID":"7272109d-9e39-4e5b-ba59-ff514f403aec","Type":"ContainerDied","Data":"9b34bf59c11d264bb4d90fc52baff27edb38a69ebac19578e1351543f49ec365"}
Sep 05 20:14:52.548655 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.548592    2583 scope.go:117] "RemoveContainer" containerID="9925c6014c39219963e56d6239a91f45b5709e593c7c56a633ebfaf2539be242"
Sep 05 20:14:52.550120 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.550081    2583 generic.go:356] "Generic (PLEG): container finished" podID="dc3ec0e0-57b6-4289-910c-6cf10b36102b" containerID="4c3df131b361c31ad4482abaa95e02dbe354171c34ab951b24d28547edf75f2b" exitCode=0
Sep 05 20:14:52.550243 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.550141    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz"
Sep 05 20:14:52.550243 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.550148    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz" event={"ID":"dc3ec0e0-57b6-4289-910c-6cf10b36102b","Type":"ContainerDied","Data":"4c3df131b361c31ad4482abaa95e02dbe354171c34ab951b24d28547edf75f2b"}
Sep 05 20:14:52.550243 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.550203    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz" event={"ID":"dc3ec0e0-57b6-4289-910c-6cf10b36102b","Type":"ContainerDied","Data":"2946cf480bdeadddc5e6c538ac213f8115f08b6f8cd15d44bc8c1f1ac97b1959"}
Sep 05 20:14:52.558407 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.558387    2583 scope.go:117] "RemoveContainer" containerID="9925c6014c39219963e56d6239a91f45b5709e593c7c56a633ebfaf2539be242"
Sep 05 20:14:52.558759 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:14:52.558733    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9925c6014c39219963e56d6239a91f45b5709e593c7c56a633ebfaf2539be242\": container with ID starting with 9925c6014c39219963e56d6239a91f45b5709e593c7c56a633ebfaf2539be242 not found: ID does not exist" containerID="9925c6014c39219963e56d6239a91f45b5709e593c7c56a633ebfaf2539be242"
Sep 05 20:14:52.558834 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.558772    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9925c6014c39219963e56d6239a91f45b5709e593c7c56a633ebfaf2539be242"} err="failed to get container status \"9925c6014c39219963e56d6239a91f45b5709e593c7c56a633ebfaf2539be242\": rpc error: code = NotFound desc = could not find container \"9925c6014c39219963e56d6239a91f45b5709e593c7c56a633ebfaf2539be242\": container with ID starting with 9925c6014c39219963e56d6239a91f45b5709e593c7c56a633ebfaf2539be242 not found: ID does not exist"
Sep 05 20:14:52.558834 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.558799    2583 scope.go:117] "RemoveContainer" containerID="4c3df131b361c31ad4482abaa95e02dbe354171c34ab951b24d28547edf75f2b"
Sep 05 20:14:52.568463 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.568427    2583 scope.go:117] "RemoveContainer" containerID="4c3df131b361c31ad4482abaa95e02dbe354171c34ab951b24d28547edf75f2b"
Sep 05 20:14:52.568951 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:14:52.568922    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c3df131b361c31ad4482abaa95e02dbe354171c34ab951b24d28547edf75f2b\": container with ID starting with 4c3df131b361c31ad4482abaa95e02dbe354171c34ab951b24d28547edf75f2b not found: ID does not exist" containerID="4c3df131b361c31ad4482abaa95e02dbe354171c34ab951b24d28547edf75f2b"
Sep 05 20:14:52.569128 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.568960    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c3df131b361c31ad4482abaa95e02dbe354171c34ab951b24d28547edf75f2b"} err="failed to get container status \"4c3df131b361c31ad4482abaa95e02dbe354171c34ab951b24d28547edf75f2b\": rpc error: code = NotFound desc = could not find container \"4c3df131b361c31ad4482abaa95e02dbe354171c34ab951b24d28547edf75f2b\": container with ID starting with 4c3df131b361c31ad4482abaa95e02dbe354171c34ab951b24d28547edf75f2b not found: ID does not exist"
Sep 05 20:14:52.572440 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.572417    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k"]
Sep 05 20:14:52.576442 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.576418    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0cc31a-6645d58c8c-b4j6k"]
Sep 05 20:14:52.587216 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.587194    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz"]
Sep 05 20:14:52.591821 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:52.591798    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0cc31a-7dd86d58-kx8wz"]
Sep 05 20:14:53.435076 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:53.435040    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7272109d-9e39-4e5b-ba59-ff514f403aec" path="/var/lib/kubelet/pods/7272109d-9e39-4e5b-ba59-ff514f403aec/volumes"
Sep 05 20:14:53.435503 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:14:53.435478    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc3ec0e0-57b6-4289-910c-6cf10b36102b" path="/var/lib/kubelet/pods/dc3ec0e0-57b6-4289-910c-6cf10b36102b/volumes"
Sep 05 20:15:00.135794 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.135758    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29810655-xwprn"]
Sep 05 20:15:00.136194 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136142    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12890272-a09c-4655-a206-464c55195412" containerName="istio-proxy"
Sep 05 20:15:00.136194 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136160    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="12890272-a09c-4655-a206-464c55195412" containerName="istio-proxy"
Sep 05 20:15:00.136194 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136171    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12890272-a09c-4655-a206-464c55195412" containerName="istio-proxy"
Sep 05 20:15:00.136194 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136179    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="12890272-a09c-4655-a206-464c55195412" containerName="istio-proxy"
Sep 05 20:15:00.136194 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136193    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a86b2193-95ed-422f-9097-3bd6378b5e99" containerName="payload-pre-processing"
Sep 05 20:15:00.136365 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136200    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a86b2193-95ed-422f-9097-3bd6378b5e99" containerName="payload-pre-processing"
Sep 05 20:15:00.136365 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136206    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4cfd886-4a10-4b97-b3bf-e7bd86e8b740" containerName="payload-pre-processing"
Sep 05 20:15:00.136365 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136214    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4cfd886-4a10-4b97-b3bf-e7bd86e8b740" containerName="payload-pre-processing"
Sep 05 20:15:00.136365 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136223    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7272109d-9e39-4e5b-ba59-ff514f403aec" containerName="payload-processing"
Sep 05 20:15:00.136365 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136231    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7272109d-9e39-4e5b-ba59-ff514f403aec" containerName="payload-processing"
Sep 05 20:15:00.136365 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136236    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c614a8a-83e2-4e79-a921-83e3dd9afbe0" containerName="payload-processing"
Sep 05 20:15:00.136365 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136242    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c614a8a-83e2-4e79-a921-83e3dd9afbe0" containerName="payload-processing"
Sep 05 20:15:00.136365 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136250    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="067dea99-d8a2-4a9e-a804-f24b6d5421e2" containerName="payload-pre-processing"
Sep 05 20:15:00.136365 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136255    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="067dea99-d8a2-4a9e-a804-f24b6d5421e2" containerName="payload-pre-processing"
Sep 05 20:15:00.136365 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136263    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be0ebded-91d9-40b0-87cd-52a3a9093edd" containerName="payload-processing"
Sep 05 20:15:00.136365 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136268    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="be0ebded-91d9-40b0-87cd-52a3a9093edd" containerName="payload-processing"
Sep 05 20:15:00.136365 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136274    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67dc0206-4c02-45ff-85c2-26896c0102b1" containerName="revoke-keys"
Sep 05 20:15:00.136365 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136279    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="67dc0206-4c02-45ff-85c2-26896c0102b1" containerName="revoke-keys"
Sep 05 20:15:00.136365 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136290    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99c1d1ac-b6c8-4047-83e8-168b8b94fc56" containerName="payload-processing"
Sep 05 20:15:00.136365 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136296    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="99c1d1ac-b6c8-4047-83e8-168b8b94fc56" containerName="payload-processing"
Sep 05 20:15:00.136365 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136301    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc3ec0e0-57b6-4289-910c-6cf10b36102b" containerName="payload-pre-processing"
Sep 05 20:15:00.136365 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136306    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc3ec0e0-57b6-4289-910c-6cf10b36102b" containerName="payload-pre-processing"
Sep 05 20:15:00.136875 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136383    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc3ec0e0-57b6-4289-910c-6cf10b36102b" containerName="payload-pre-processing"
Sep 05 20:15:00.136875 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136390    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7272109d-9e39-4e5b-ba59-ff514f403aec" containerName="payload-processing"
Sep 05 20:15:00.136875 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136397    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="12890272-a09c-4655-a206-464c55195412" containerName="istio-proxy"
Sep 05 20:15:00.136875 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136403    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c614a8a-83e2-4e79-a921-83e3dd9afbe0" containerName="payload-processing"
Sep 05 20:15:00.136875 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136407    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="be0ebded-91d9-40b0-87cd-52a3a9093edd" containerName="payload-processing"
Sep 05 20:15:00.136875 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136413    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="67dc0206-4c02-45ff-85c2-26896c0102b1" containerName="revoke-keys"
Sep 05 20:15:00.136875 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136418    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="12890272-a09c-4655-a206-464c55195412" containerName="istio-proxy"
Sep 05 20:15:00.136875 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136422    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="99c1d1ac-b6c8-4047-83e8-168b8b94fc56" containerName="payload-processing"
Sep 05 20:15:00.136875 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136426    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4cfd886-4a10-4b97-b3bf-e7bd86e8b740" containerName="payload-pre-processing"
Sep 05 20:15:00.136875 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136431    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="067dea99-d8a2-4a9e-a804-f24b6d5421e2" containerName="payload-pre-processing"
Sep 05 20:15:00.136875 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.136436    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a86b2193-95ed-422f-9097-3bd6378b5e99" containerName="payload-pre-processing"
Sep 05 20:15:00.139468 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.139451    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29810655-xwprn"
Sep 05 20:15:00.141508 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.141482    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zx89\""
Sep 05 20:15:00.146623 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.146600    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29810655-xwprn"]
Sep 05 20:15:00.150585 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.150565    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29810655-xwprn"
Sep 05 20:15:00.422928 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.422899    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29810655-xwprn"]
Sep 05 20:15:00.424868 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:15:00.424835    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce4899f7_bc8c_4015_bcdb_f6603a4a7f7f.slice/crio-1daff7994f2fe63e2c3b8ddc35e38baa3448e9a25220fafba781ab878618d881 WatchSource:0}: Error finding container 1daff7994f2fe63e2c3b8ddc35e38baa3448e9a25220fafba781ab878618d881: Status 404 returned error can't find the container with id 1daff7994f2fe63e2c3b8ddc35e38baa3448e9a25220fafba781ab878618d881
Sep 05 20:15:00.603123 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:00.603084    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29810655-xwprn" event={"ID":"ce4899f7-bc8c-4015-bcdb-f6603a4a7f7f","Type":"ContainerStarted","Data":"1daff7994f2fe63e2c3b8ddc35e38baa3448e9a25220fafba781ab878618d881"}
Sep 05 20:15:01.610548 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:01.610513    2583 generic.go:356] "Generic (PLEG): container finished" podID="ce4899f7-bc8c-4015-bcdb-f6603a4a7f7f" containerID="ecf35e3a95589e58b09a655312667d32e50b1609c6762be2a5add91a273929e8" exitCode=0
Sep 05 20:15:01.610913 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:01.610598    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29810655-xwprn" event={"ID":"ce4899f7-bc8c-4015-bcdb-f6603a4a7f7f","Type":"ContainerDied","Data":"ecf35e3a95589e58b09a655312667d32e50b1609c6762be2a5add91a273929e8"}
Sep 05 20:15:02.756709 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:02.756678    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29810655-xwprn"
Sep 05 20:15:03.623863 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:03.623832    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29810655-xwprn"
Sep 05 20:15:03.624032 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:03.623883    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29810655-xwprn" event={"ID":"ce4899f7-bc8c-4015-bcdb-f6603a4a7f7f","Type":"ContainerDied","Data":"1daff7994f2fe63e2c3b8ddc35e38baa3448e9a25220fafba781ab878618d881"}
Sep 05 20:15:03.624032 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:15:03.623927    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1daff7994f2fe63e2c3b8ddc35e38baa3448e9a25220fafba781ab878618d881"
Sep 05 20:16:22.007292 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:16:22.007217    2583 scope.go:117] "RemoveContainer" containerID="3a654a8d67a0c36752164ea88dca3a096f80bbc36e5aab3d5f7f70cb0f0a6e46"
Sep 05 20:16:22.015577 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:16:22.015559    2583 scope.go:117] "RemoveContainer" containerID="2acd4002a6fa805571ac7c95231d840f1eb3a7139decbf7315714265cba17b9e"
Sep 05 20:17:22.039300 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:17:22.039270    2583 scope.go:117] "RemoveContainer" containerID="1539dc7b80e9edca29b448c454be7c6cdc2e0db2842408a56bc9916e3caa3365"
Sep 05 20:17:22.047828 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:17:22.047807    2583 scope.go:117] "RemoveContainer" containerID="ab4c75a2353ab6ce330340743ef4b65bd01807f6ca6ea05cc0b47e8cf57e57e2"
Sep 05 20:17:41.086435 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:17:41.086388    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-kg8m8"]
Sep 05 20:17:41.087023 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:17:41.086708    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-kg8m8" podUID="3259161d-2c56-4705-aca8-ff45669a1915" containerName="manager" containerID="cri-o://55756185eff23eeaabaeb5b7b6c6ffe1eb461d42e11239631d565d9b5c068f54" gracePeriod=10
Sep 05 20:17:42.049932 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:17:42.049907    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-kg8m8"
Sep 05 20:17:42.097383 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:17:42.097348    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3259161d-2c56-4705-aca8-ff45669a1915-extensions-socket-volume\") pod \"3259161d-2c56-4705-aca8-ff45669a1915\" (UID: \"3259161d-2c56-4705-aca8-ff45669a1915\") "
Sep 05 20:17:42.097810 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:17:42.097440    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tcdd\" (UniqueName: \"kubernetes.io/projected/3259161d-2c56-4705-aca8-ff45669a1915-kube-api-access-4tcdd\") pod \"3259161d-2c56-4705-aca8-ff45669a1915\" (UID: \"3259161d-2c56-4705-aca8-ff45669a1915\") "
Sep 05 20:17:42.097810 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:17:42.097779    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3259161d-2c56-4705-aca8-ff45669a1915-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3259161d-2c56-4705-aca8-ff45669a1915" (UID: "3259161d-2c56-4705-aca8-ff45669a1915"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:17:42.099751 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:17:42.099726    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3259161d-2c56-4705-aca8-ff45669a1915-kube-api-access-4tcdd" (OuterVolumeSpecName: "kube-api-access-4tcdd") pod "3259161d-2c56-4705-aca8-ff45669a1915" (UID: "3259161d-2c56-4705-aca8-ff45669a1915"). InnerVolumeSpecName "kube-api-access-4tcdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:17:42.198806 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:17:42.198730    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tcdd\" (UniqueName: \"kubernetes.io/projected/3259161d-2c56-4705-aca8-ff45669a1915-kube-api-access-4tcdd\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:17:42.198806 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:17:42.198755    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3259161d-2c56-4705-aca8-ff45669a1915-extensions-socket-volume\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:17:42.534692 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:17:42.534606    2583 generic.go:356] "Generic (PLEG): container finished" podID="3259161d-2c56-4705-aca8-ff45669a1915" containerID="55756185eff23eeaabaeb5b7b6c6ffe1eb461d42e11239631d565d9b5c068f54" exitCode=0
Sep 05 20:17:42.534692 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:17:42.534649    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-kg8m8"
Sep 05 20:17:42.534886 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:17:42.534699    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-kg8m8" event={"ID":"3259161d-2c56-4705-aca8-ff45669a1915","Type":"ContainerDied","Data":"55756185eff23eeaabaeb5b7b6c6ffe1eb461d42e11239631d565d9b5c068f54"}
Sep 05 20:17:42.534886 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:17:42.534755    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-kg8m8" event={"ID":"3259161d-2c56-4705-aca8-ff45669a1915","Type":"ContainerDied","Data":"d06aab4a8d88959e0fcf09b4dc631424ef8f3985331b0befea30c34ace168d89"}
Sep 05 20:17:42.534886 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:17:42.534781    2583 scope.go:117] "RemoveContainer" containerID="55756185eff23eeaabaeb5b7b6c6ffe1eb461d42e11239631d565d9b5c068f54"
Sep 05 20:17:42.545512 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:17:42.545492    2583 scope.go:117] "RemoveContainer" containerID="55756185eff23eeaabaeb5b7b6c6ffe1eb461d42e11239631d565d9b5c068f54"
Sep 05 20:17:42.545778 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:17:42.545758    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55756185eff23eeaabaeb5b7b6c6ffe1eb461d42e11239631d565d9b5c068f54\": container with ID starting with 55756185eff23eeaabaeb5b7b6c6ffe1eb461d42e11239631d565d9b5c068f54 not found: ID does not exist" containerID="55756185eff23eeaabaeb5b7b6c6ffe1eb461d42e11239631d565d9b5c068f54"
Sep 05 20:17:42.545852 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:17:42.545786    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55756185eff23eeaabaeb5b7b6c6ffe1eb461d42e11239631d565d9b5c068f54"} err="failed to get container status \"55756185eff23eeaabaeb5b7b6c6ffe1eb461d42e11239631d565d9b5c068f54\": rpc error: code = NotFound desc = could not find container \"55756185eff23eeaabaeb5b7b6c6ffe1eb461d42e11239631d565d9b5c068f54\": container with ID starting with 55756185eff23eeaabaeb5b7b6c6ffe1eb461d42e11239631d565d9b5c068f54 not found: ID does not exist"
Sep 05 20:17:42.558220 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:17:42.558198    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-kg8m8"]
Sep 05 20:17:42.569601 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:17:42.569580    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-kg8m8"]
Sep 05 20:17:43.434679 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:17:43.434645    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3259161d-2c56-4705-aca8-ff45669a1915" path="/var/lib/kubelet/pods/3259161d-2c56-4705-aca8-ff45669a1915/volumes"
Sep 05 20:18:21.639643 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:21.639613    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf9tj_f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b/ovn-acl-logging/0.log"
Sep 05 20:18:21.652231 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:21.652206    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mf9tj_f2e5cbc4-f09e-417d-a9f9-22fc1b1c239b/ovn-acl-logging/0.log"
Sep 05 20:18:41.768093 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:41.768061    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-6pzml"]
Sep 05 20:18:41.768563 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:41.768441    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3259161d-2c56-4705-aca8-ff45669a1915" containerName="manager"
Sep 05 20:18:41.768563 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:41.768454    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3259161d-2c56-4705-aca8-ff45669a1915" containerName="manager"
Sep 05 20:18:41.768563 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:41.768475    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce4899f7-bc8c-4015-bcdb-f6603a4a7f7f" containerName="cleanup"
Sep 05 20:18:41.768563 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:41.768480    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce4899f7-bc8c-4015-bcdb-f6603a4a7f7f" containerName="cleanup"
Sep 05 20:18:41.768563 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:41.768534    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3259161d-2c56-4705-aca8-ff45669a1915" containerName="manager"
Sep 05 20:18:41.768563 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:41.768542    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce4899f7-bc8c-4015-bcdb-f6603a4a7f7f" containerName="cleanup"
Sep 05 20:18:41.771501 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:41.771483    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-6pzml"
Sep 05 20:18:41.773525 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:41.773505    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tq2z7\""
Sep 05 20:18:41.782743 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:41.782718    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-6pzml"]
Sep 05 20:18:41.824675 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:41.824634    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqv2x\" (UniqueName: \"kubernetes.io/projected/ec15b1dd-1ce4-4134-a56f-49e8b8c84ca9-kube-api-access-mqv2x\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-6pzml\" (UID: \"ec15b1dd-1ce4-4134-a56f-49e8b8c84ca9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-6pzml"
Sep 05 20:18:41.824842 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:41.824688    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec15b1dd-1ce4-4134-a56f-49e8b8c84ca9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-6pzml\" (UID: \"ec15b1dd-1ce4-4134-a56f-49e8b8c84ca9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-6pzml"
Sep 05 20:18:41.926010 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:41.925968    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec15b1dd-1ce4-4134-a56f-49e8b8c84ca9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-6pzml\" (UID: \"ec15b1dd-1ce4-4134-a56f-49e8b8c84ca9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-6pzml"
Sep 05 20:18:41.926220 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:41.926074    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqv2x\" (UniqueName: \"kubernetes.io/projected/ec15b1dd-1ce4-4134-a56f-49e8b8c84ca9-kube-api-access-mqv2x\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-6pzml\" (UID: \"ec15b1dd-1ce4-4134-a56f-49e8b8c84ca9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-6pzml"
Sep 05 20:18:41.926404 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:41.926378    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec15b1dd-1ce4-4134-a56f-49e8b8c84ca9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-6pzml\" (UID: \"ec15b1dd-1ce4-4134-a56f-49e8b8c84ca9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-6pzml"
Sep 05 20:18:41.933741 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:41.933715    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqv2x\" (UniqueName: \"kubernetes.io/projected/ec15b1dd-1ce4-4134-a56f-49e8b8c84ca9-kube-api-access-mqv2x\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-6pzml\" (UID: \"ec15b1dd-1ce4-4134-a56f-49e8b8c84ca9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-6pzml"
Sep 05 20:18:42.082522 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:42.082473    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-6pzml"
Sep 05 20:18:42.356013 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:42.355987    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-6pzml"]
Sep 05 20:18:42.358148 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:18:42.358097    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec15b1dd_1ce4_4134_a56f_49e8b8c84ca9.slice/crio-1d29ed18569ae508af4b94763595778bf2ae6bebf6b76a8c6a5ecb1bf1016cb8 WatchSource:0}: Error finding container 1d29ed18569ae508af4b94763595778bf2ae6bebf6b76a8c6a5ecb1bf1016cb8: Status 404 returned error can't find the container with id 1d29ed18569ae508af4b94763595778bf2ae6bebf6b76a8c6a5ecb1bf1016cb8
Sep 05 20:18:42.872698 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:42.872659    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-6pzml" event={"ID":"ec15b1dd-1ce4-4134-a56f-49e8b8c84ca9","Type":"ContainerStarted","Data":"c20872f693286a61d037554680c0a64f26d9cdb4bd5fcd2abfa7fe5887dc4cff"}
Sep 05 20:18:42.873084 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:42.872716    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-6pzml" event={"ID":"ec15b1dd-1ce4-4134-a56f-49e8b8c84ca9","Type":"ContainerStarted","Data":"1d29ed18569ae508af4b94763595778bf2ae6bebf6b76a8c6a5ecb1bf1016cb8"}
Sep 05 20:18:42.873084 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:42.872772    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-6pzml"
Sep 05 20:18:42.891967 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:42.891913    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-6pzml" podStartSLOduration=1.891898575 podStartE2EDuration="1.891898575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:18:42.88846715 +0000 UTC m=+2121.918099054" watchObservedRunningTime="2026-09-05 20:18:42.891898575 +0000 UTC m=+2121.921530480"
Sep 05 20:18:53.881405 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:18:53.881367    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-6pzml"
Sep 05 20:19:22.084844 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:22.084770    2583 scope.go:117] "RemoveContainer" containerID="5a4aaa67cb7bc9097c37e94a2c227ef77b805fe8819cc7cc9d1c25f935c7b46d"
Sep 05 20:19:23.221947 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:23.221910    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"]
Sep 05 20:19:23.227497 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:23.227472    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:19:23.229591 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:23.229537    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-07fb4624-kserve-self-signed-certs\""
Sep 05 20:19:23.234577 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:23.234546    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"]
Sep 05 20:19:23.290436 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:23.290401    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2faded2-d1d3-405a-8de5-377ee528041d-tmp-dir\") pod \"e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww\" (UID: \"e2faded2-d1d3-405a-8de5-377ee528041d\") " pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:19:23.290616 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:23.290446    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2faded2-d1d3-405a-8de5-377ee528041d-tls-certs\") pod \"e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww\" (UID: \"e2faded2-d1d3-405a-8de5-377ee528041d\") " pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:19:23.290616 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:23.290480    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njt7q\" (UniqueName: \"kubernetes.io/projected/e2faded2-d1d3-405a-8de5-377ee528041d-kube-api-access-njt7q\") pod \"e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww\" (UID: \"e2faded2-d1d3-405a-8de5-377ee528041d\") " pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:19:23.290616 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:23.290502    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2faded2-d1d3-405a-8de5-377ee528041d-model-cache\") pod \"e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww\" (UID: \"e2faded2-d1d3-405a-8de5-377ee528041d\") " pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:19:23.290616 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:23.290520    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2faded2-d1d3-405a-8de5-377ee528041d-home\") pod \"e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww\" (UID: \"e2faded2-d1d3-405a-8de5-377ee528041d\") " pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:19:23.290616 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:23.290535    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2faded2-d1d3-405a-8de5-377ee528041d-dshm\") pod \"e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww\" (UID: \"e2faded2-d1d3-405a-8de5-377ee528041d\") " pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:19:23.391316 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:23.391280    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2faded2-d1d3-405a-8de5-377ee528041d-tls-certs\") pod \"e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww\" (UID: \"e2faded2-d1d3-405a-8de5-377ee528041d\") " pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:19:23.391489 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:23.391334    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njt7q\" (UniqueName: \"kubernetes.io/projected/e2faded2-d1d3-405a-8de5-377ee528041d-kube-api-access-njt7q\") pod \"e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww\" (UID: \"e2faded2-d1d3-405a-8de5-377ee528041d\") " pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:19:23.391489 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:23.391366    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2faded2-d1d3-405a-8de5-377ee528041d-model-cache\") pod \"e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww\" (UID: \"e2faded2-d1d3-405a-8de5-377ee528041d\") " pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:19:23.391489 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:23.391388    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2faded2-d1d3-405a-8de5-377ee528041d-home\") pod \"e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww\" (UID: \"e2faded2-d1d3-405a-8de5-377ee528041d\") " pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:19:23.391489 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:23.391402    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2faded2-d1d3-405a-8de5-377ee528041d-dshm\") pod \"e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww\" (UID: \"e2faded2-d1d3-405a-8de5-377ee528041d\") " pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:19:23.391489 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:23.391470    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2faded2-d1d3-405a-8de5-377ee528041d-tmp-dir\") pod \"e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww\" (UID: \"e2faded2-d1d3-405a-8de5-377ee528041d\") " pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:19:23.391804 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:23.391781    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2faded2-d1d3-405a-8de5-377ee528041d-home\") pod \"e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww\" (UID: \"e2faded2-d1d3-405a-8de5-377ee528041d\") " pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:19:23.391878 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:23.391832    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2faded2-d1d3-405a-8de5-377ee528041d-tmp-dir\") pod \"e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww\" (UID: \"e2faded2-d1d3-405a-8de5-377ee528041d\") " pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:19:23.391878 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:23.391872    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2faded2-d1d3-405a-8de5-377ee528041d-model-cache\") pod \"e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww\" (UID: \"e2faded2-d1d3-405a-8de5-377ee528041d\") " pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:19:23.393653 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:23.393627    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2faded2-d1d3-405a-8de5-377ee528041d-dshm\") pod \"e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww\" (UID: \"e2faded2-d1d3-405a-8de5-377ee528041d\") " pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:19:23.393920 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:23.393900    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2faded2-d1d3-405a-8de5-377ee528041d-tls-certs\") pod \"e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww\" (UID: \"e2faded2-d1d3-405a-8de5-377ee528041d\") " pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:19:23.398525 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:23.398506    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njt7q\" (UniqueName: \"kubernetes.io/projected/e2faded2-d1d3-405a-8de5-377ee528041d-kube-api-access-njt7q\") pod \"e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww\" (UID: \"e2faded2-d1d3-405a-8de5-377ee528041d\") " pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:19:23.539660 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:23.539568    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:19:23.808435 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:23.808400    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"]
Sep 05 20:19:23.809278 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:19:23.809252    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2faded2_d1d3_405a_8de5_377ee528041d.slice/crio-a10721ba50e7a3e5307e5e8e655fcd8ef91dfaa725c1c424ee257ae45b69bd86 WatchSource:0}: Error finding container a10721ba50e7a3e5307e5e8e655fcd8ef91dfaa725c1c424ee257ae45b69bd86: Status 404 returned error can't find the container with id a10721ba50e7a3e5307e5e8e655fcd8ef91dfaa725c1c424ee257ae45b69bd86
Sep 05 20:19:24.114767 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:24.114730    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww" event={"ID":"e2faded2-d1d3-405a-8de5-377ee528041d","Type":"ContainerStarted","Data":"14fd0a2dd123097ccfeb143145613955429d3e4e65c15c5b4540cd779e425522"}
Sep 05 20:19:24.114767 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:24.114775    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww" event={"ID":"e2faded2-d1d3-405a-8de5-377ee528041d","Type":"ContainerStarted","Data":"a10721ba50e7a3e5307e5e8e655fcd8ef91dfaa725c1c424ee257ae45b69bd86"}
Sep 05 20:19:24.129579 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:24.129528    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww" podStartSLOduration=1.129512272 podStartE2EDuration="1.129512272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:19:24.128084848 +0000 UTC m=+2163.157716754" watchObservedRunningTime="2026-09-05 20:19:24.129512272 +0000 UTC m=+2163.159144177"
Sep 05 20:19:33.539679 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:33.539640    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:19:33.540345 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:33.539698    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:19:33.552675 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:33.552649    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:19:34.183443 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:34.183400    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:19:36.817795 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:36.817757    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"]
Sep 05 20:19:36.821483 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:36.821465    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:19:36.823436 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:36.823414    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-07fb4624-kserve-self-signed-certs\""
Sep 05 20:19:36.828316 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:36.828292    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"]
Sep 05 20:19:36.916948 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:36.916904    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvkxk\" (UniqueName: \"kubernetes.io/projected/ba64c0ce-499f-4940-9060-7d1e4b187ad9-kube-api-access-pvkxk\") pod \"e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9\" (UID: \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\") " pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:19:36.917163 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:36.916961    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba64c0ce-499f-4940-9060-7d1e4b187ad9-model-cache\") pod \"e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9\" (UID: \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\") " pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:19:36.917163 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:36.917027    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba64c0ce-499f-4940-9060-7d1e4b187ad9-dshm\") pod \"e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9\" (UID: \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\") " pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:19:36.917163 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:36.917057    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba64c0ce-499f-4940-9060-7d1e4b187ad9-tls-certs\") pod \"e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9\" (UID: \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\") " pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:19:36.917163 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:36.917093    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba64c0ce-499f-4940-9060-7d1e4b187ad9-home\") pod \"e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9\" (UID: \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\") " pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:19:36.917163 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:36.917130    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba64c0ce-499f-4940-9060-7d1e4b187ad9-tmp-dir\") pod \"e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9\" (UID: \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\") " pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:19:37.018441 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:37.018400    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvkxk\" (UniqueName: \"kubernetes.io/projected/ba64c0ce-499f-4940-9060-7d1e4b187ad9-kube-api-access-pvkxk\") pod \"e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9\" (UID: \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\") " pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:19:37.018619 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:37.018498    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba64c0ce-499f-4940-9060-7d1e4b187ad9-model-cache\") pod \"e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9\" (UID: \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\") " pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:19:37.018619 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:37.018521    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba64c0ce-499f-4940-9060-7d1e4b187ad9-dshm\") pod \"e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9\" (UID: \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\") " pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:19:37.018619 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:37.018538    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba64c0ce-499f-4940-9060-7d1e4b187ad9-tls-certs\") pod \"e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9\" (UID: \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\") " pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:19:37.018619 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:37.018593    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba64c0ce-499f-4940-9060-7d1e4b187ad9-home\") pod \"e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9\" (UID: \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\") " pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:19:37.018619 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:37.018619    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba64c0ce-499f-4940-9060-7d1e4b187ad9-tmp-dir\") pod \"e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9\" (UID: \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\") " pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:19:37.018995 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:37.018969    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba64c0ce-499f-4940-9060-7d1e4b187ad9-model-cache\") pod \"e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9\" (UID: \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\") " pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:19:37.018995 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:37.018986    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba64c0ce-499f-4940-9060-7d1e4b187ad9-home\") pod \"e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9\" (UID: \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\") " pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:19:37.019174 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:37.019024    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba64c0ce-499f-4940-9060-7d1e4b187ad9-tmp-dir\") pod \"e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9\" (UID: \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\") " pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:19:37.020785 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:37.020760    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba64c0ce-499f-4940-9060-7d1e4b187ad9-dshm\") pod \"e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9\" (UID: \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\") " pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:19:37.021027 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:37.021008    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba64c0ce-499f-4940-9060-7d1e4b187ad9-tls-certs\") pod \"e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9\" (UID: \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\") " pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:19:37.025213 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:37.025192    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvkxk\" (UniqueName: \"kubernetes.io/projected/ba64c0ce-499f-4940-9060-7d1e4b187ad9-kube-api-access-pvkxk\") pod \"e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9\" (UID: \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\") " pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:19:37.133228 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:37.133190    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:19:37.405977 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:37.405953    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"]
Sep 05 20:19:37.407893 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:19:37.407861    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba64c0ce_499f_4940_9060_7d1e4b187ad9.slice/crio-4b8d2d7feac0d2a3ae75fa4f470b2d3b91b0b51f1897887a3e5a76f8706aa017 WatchSource:0}: Error finding container 4b8d2d7feac0d2a3ae75fa4f470b2d3b91b0b51f1897887a3e5a76f8706aa017: Status 404 returned error can't find the container with id 4b8d2d7feac0d2a3ae75fa4f470b2d3b91b0b51f1897887a3e5a76f8706aa017
Sep 05 20:19:38.199433 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:38.199393    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9" event={"ID":"ba64c0ce-499f-4940-9060-7d1e4b187ad9","Type":"ContainerStarted","Data":"a6d382c766a36e744aab37001065b8fddd6a1d92d25ba35d510f04199d013923"}
Sep 05 20:19:38.199433 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:38.199440    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9" event={"ID":"ba64c0ce-499f-4940-9060-7d1e4b187ad9","Type":"ContainerStarted","Data":"4b8d2d7feac0d2a3ae75fa4f470b2d3b91b0b51f1897887a3e5a76f8706aa017"}
Sep 05 20:19:38.213566 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:38.213485    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9" podStartSLOduration=2.21346475 podStartE2EDuration="2.21346475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:19:38.212888854 +0000 UTC m=+2177.242520774" watchObservedRunningTime="2026-09-05 20:19:38.21346475 +0000 UTC m=+2177.243096656"
Sep 05 20:19:47.133476 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:47.133441    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:19:47.133476 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:47.133483    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:19:47.146031 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:47.146004    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:19:47.259926 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:47.259896    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:19:59.920100 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:19:59.920065    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"]
Sep 05 20:20:01.128373 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:01.128332    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"]
Sep 05 20:20:14.939927 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:14.939858    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww" podUID="e2faded2-d1d3-405a-8de5-377ee528041d" containerName="main" containerID="cri-o://14fd0a2dd123097ccfeb143145613955429d3e4e65c15c5b4540cd779e425522" gracePeriod=45
Sep 05 20:20:15.186196 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.186167    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:20:15.282779 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.282701    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2faded2-d1d3-405a-8de5-377ee528041d-tls-certs\") pod \"e2faded2-d1d3-405a-8de5-377ee528041d\" (UID: \"e2faded2-d1d3-405a-8de5-377ee528041d\") "
Sep 05 20:20:15.282779 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.282738    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2faded2-d1d3-405a-8de5-377ee528041d-home\") pod \"e2faded2-d1d3-405a-8de5-377ee528041d\" (UID: \"e2faded2-d1d3-405a-8de5-377ee528041d\") "
Sep 05 20:20:15.282779 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.282763    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njt7q\" (UniqueName: \"kubernetes.io/projected/e2faded2-d1d3-405a-8de5-377ee528041d-kube-api-access-njt7q\") pod \"e2faded2-d1d3-405a-8de5-377ee528041d\" (UID: \"e2faded2-d1d3-405a-8de5-377ee528041d\") "
Sep 05 20:20:15.283065 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.282791    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2faded2-d1d3-405a-8de5-377ee528041d-model-cache\") pod \"e2faded2-d1d3-405a-8de5-377ee528041d\" (UID: \"e2faded2-d1d3-405a-8de5-377ee528041d\") "
Sep 05 20:20:15.283065 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.282817    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2faded2-d1d3-405a-8de5-377ee528041d-dshm\") pod \"e2faded2-d1d3-405a-8de5-377ee528041d\" (UID: \"e2faded2-d1d3-405a-8de5-377ee528041d\") "
Sep 05 20:20:15.283065 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.282912    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2faded2-d1d3-405a-8de5-377ee528041d-tmp-dir\") pod \"e2faded2-d1d3-405a-8de5-377ee528041d\" (UID: \"e2faded2-d1d3-405a-8de5-377ee528041d\") "
Sep 05 20:20:15.283248 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.283041    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2faded2-d1d3-405a-8de5-377ee528041d-home" (OuterVolumeSpecName: "home") pod "e2faded2-d1d3-405a-8de5-377ee528041d" (UID: "e2faded2-d1d3-405a-8de5-377ee528041d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:20:15.283248 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.283146    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2faded2-d1d3-405a-8de5-377ee528041d-model-cache" (OuterVolumeSpecName: "model-cache") pod "e2faded2-d1d3-405a-8de5-377ee528041d" (UID: "e2faded2-d1d3-405a-8de5-377ee528041d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:20:15.283248 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.283203    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2faded2-d1d3-405a-8de5-377ee528041d-home\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:20:15.283363 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.283277    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2faded2-d1d3-405a-8de5-377ee528041d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e2faded2-d1d3-405a-8de5-377ee528041d" (UID: "e2faded2-d1d3-405a-8de5-377ee528041d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:20:15.284903 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.284877    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2faded2-d1d3-405a-8de5-377ee528041d-dshm" (OuterVolumeSpecName: "dshm") pod "e2faded2-d1d3-405a-8de5-377ee528041d" (UID: "e2faded2-d1d3-405a-8de5-377ee528041d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:20:15.285006 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.284882    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2faded2-d1d3-405a-8de5-377ee528041d-kube-api-access-njt7q" (OuterVolumeSpecName: "kube-api-access-njt7q") pod "e2faded2-d1d3-405a-8de5-377ee528041d" (UID: "e2faded2-d1d3-405a-8de5-377ee528041d"). InnerVolumeSpecName "kube-api-access-njt7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:20:15.285006 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.284911    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2faded2-d1d3-405a-8de5-377ee528041d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e2faded2-d1d3-405a-8de5-377ee528041d" (UID: "e2faded2-d1d3-405a-8de5-377ee528041d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:20:15.384302 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.384264    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2faded2-d1d3-405a-8de5-377ee528041d-model-cache\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:20:15.384302 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.384294    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2faded2-d1d3-405a-8de5-377ee528041d-dshm\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:20:15.384302 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.384302    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2faded2-d1d3-405a-8de5-377ee528041d-tmp-dir\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:20:15.384302 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.384310    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2faded2-d1d3-405a-8de5-377ee528041d-tls-certs\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:20:15.384582 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.384318    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njt7q\" (UniqueName: \"kubernetes.io/projected/e2faded2-d1d3-405a-8de5-377ee528041d-kube-api-access-njt7q\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:20:15.405944 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.405912    2583 generic.go:356] "Generic (PLEG): container finished" podID="e2faded2-d1d3-405a-8de5-377ee528041d" containerID="14fd0a2dd123097ccfeb143145613955429d3e4e65c15c5b4540cd779e425522" exitCode=0
Sep 05 20:20:15.406091 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.405957    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"
Sep 05 20:20:15.406091 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.405974    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww" event={"ID":"e2faded2-d1d3-405a-8de5-377ee528041d","Type":"ContainerDied","Data":"14fd0a2dd123097ccfeb143145613955429d3e4e65c15c5b4540cd779e425522"}
Sep 05 20:20:15.406091 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.406004    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww" event={"ID":"e2faded2-d1d3-405a-8de5-377ee528041d","Type":"ContainerDied","Data":"a10721ba50e7a3e5307e5e8e655fcd8ef91dfaa725c1c424ee257ae45b69bd86"}
Sep 05 20:20:15.406091 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.406019    2583 scope.go:117] "RemoveContainer" containerID="14fd0a2dd123097ccfeb143145613955429d3e4e65c15c5b4540cd779e425522"
Sep 05 20:20:15.415274 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.415256    2583 scope.go:117] "RemoveContainer" containerID="14fd0a2dd123097ccfeb143145613955429d3e4e65c15c5b4540cd779e425522"
Sep 05 20:20:15.415555 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:20:15.415534    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14fd0a2dd123097ccfeb143145613955429d3e4e65c15c5b4540cd779e425522\": container with ID starting with 14fd0a2dd123097ccfeb143145613955429d3e4e65c15c5b4540cd779e425522 not found: ID does not exist" containerID="14fd0a2dd123097ccfeb143145613955429d3e4e65c15c5b4540cd779e425522"
Sep 05 20:20:15.415611 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.415562    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14fd0a2dd123097ccfeb143145613955429d3e4e65c15c5b4540cd779e425522"} err="failed to get container status \"14fd0a2dd123097ccfeb143145613955429d3e4e65c15c5b4540cd779e425522\": rpc error: code = NotFound desc = could not find container \"14fd0a2dd123097ccfeb143145613955429d3e4e65c15c5b4540cd779e425522\": container with ID starting with 14fd0a2dd123097ccfeb143145613955429d3e4e65c15c5b4540cd779e425522 not found: ID does not exist"
Sep 05 20:20:15.426556 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.426529    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"]
Sep 05 20:20:15.434353 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:15.434335    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-07fb4624-kserve-5bbcc4759f-mp2ww"]
Sep 05 20:20:16.154747 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.154699    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9" podUID="ba64c0ce-499f-4940-9060-7d1e4b187ad9" containerName="main" containerID="cri-o://a6d382c766a36e744aab37001065b8fddd6a1d92d25ba35d510f04199d013923" gracePeriod=45
Sep 05 20:20:16.405776 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.405710    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:20:16.413172 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.413144    2583 generic.go:356] "Generic (PLEG): container finished" podID="ba64c0ce-499f-4940-9060-7d1e4b187ad9" containerID="a6d382c766a36e744aab37001065b8fddd6a1d92d25ba35d510f04199d013923" exitCode=0
Sep 05 20:20:16.413287 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.413226    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9" event={"ID":"ba64c0ce-499f-4940-9060-7d1e4b187ad9","Type":"ContainerDied","Data":"a6d382c766a36e744aab37001065b8fddd6a1d92d25ba35d510f04199d013923"}
Sep 05 20:20:16.413287 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.413233    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"
Sep 05 20:20:16.413287 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.413268    2583 scope.go:117] "RemoveContainer" containerID="a6d382c766a36e744aab37001065b8fddd6a1d92d25ba35d510f04199d013923"
Sep 05 20:20:16.413426 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.413257    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9" event={"ID":"ba64c0ce-499f-4940-9060-7d1e4b187ad9","Type":"ContainerDied","Data":"4b8d2d7feac0d2a3ae75fa4f470b2d3b91b0b51f1897887a3e5a76f8706aa017"}
Sep 05 20:20:16.422975 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.422947    2583 scope.go:117] "RemoveContainer" containerID="a6d382c766a36e744aab37001065b8fddd6a1d92d25ba35d510f04199d013923"
Sep 05 20:20:16.423312 ip-10-0-142-208 kubenswrapper[2583]: E0905 20:20:16.423288    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6d382c766a36e744aab37001065b8fddd6a1d92d25ba35d510f04199d013923\": container with ID starting with a6d382c766a36e744aab37001065b8fddd6a1d92d25ba35d510f04199d013923 not found: ID does not exist" containerID="a6d382c766a36e744aab37001065b8fddd6a1d92d25ba35d510f04199d013923"
Sep 05 20:20:16.423423 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.423324    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d382c766a36e744aab37001065b8fddd6a1d92d25ba35d510f04199d013923"} err="failed to get container status \"a6d382c766a36e744aab37001065b8fddd6a1d92d25ba35d510f04199d013923\": rpc error: code = NotFound desc = could not find container \"a6d382c766a36e744aab37001065b8fddd6a1d92d25ba35d510f04199d013923\": container with ID starting with a6d382c766a36e744aab37001065b8fddd6a1d92d25ba35d510f04199d013923 not found: ID does not exist"
Sep 05 20:20:16.494347 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.494316    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba64c0ce-499f-4940-9060-7d1e4b187ad9-model-cache\") pod \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\" (UID: \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\") "
Sep 05 20:20:16.494511 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.494366    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba64c0ce-499f-4940-9060-7d1e4b187ad9-home\") pod \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\" (UID: \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\") "
Sep 05 20:20:16.494511 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.494391    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvkxk\" (UniqueName: \"kubernetes.io/projected/ba64c0ce-499f-4940-9060-7d1e4b187ad9-kube-api-access-pvkxk\") pod \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\" (UID: \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\") "
Sep 05 20:20:16.494511 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.494416    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba64c0ce-499f-4940-9060-7d1e4b187ad9-tmp-dir\") pod \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\" (UID: \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\") "
Sep 05 20:20:16.494511 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.494439    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba64c0ce-499f-4940-9060-7d1e4b187ad9-tls-certs\") pod \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\" (UID: \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\") "
Sep 05 20:20:16.494511 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.494458    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba64c0ce-499f-4940-9060-7d1e4b187ad9-dshm\") pod \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\" (UID: \"ba64c0ce-499f-4940-9060-7d1e4b187ad9\") "
Sep 05 20:20:16.494791 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.494602    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba64c0ce-499f-4940-9060-7d1e4b187ad9-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba64c0ce-499f-4940-9060-7d1e4b187ad9" (UID: "ba64c0ce-499f-4940-9060-7d1e4b187ad9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:20:16.494791 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.494626    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba64c0ce-499f-4940-9060-7d1e4b187ad9-home" (OuterVolumeSpecName: "home") pod "ba64c0ce-499f-4940-9060-7d1e4b187ad9" (UID: "ba64c0ce-499f-4940-9060-7d1e4b187ad9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:20:16.494791 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.494665    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba64c0ce-499f-4940-9060-7d1e4b187ad9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba64c0ce-499f-4940-9060-7d1e4b187ad9" (UID: "ba64c0ce-499f-4940-9060-7d1e4b187ad9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:20:16.494791 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.494783    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba64c0ce-499f-4940-9060-7d1e4b187ad9-home\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:20:16.494962 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.494801    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba64c0ce-499f-4940-9060-7d1e4b187ad9-tmp-dir\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:20:16.494962 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.494817    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba64c0ce-499f-4940-9060-7d1e4b187ad9-model-cache\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:20:16.496665 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.496640    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba64c0ce-499f-4940-9060-7d1e4b187ad9-dshm" (OuterVolumeSpecName: "dshm") pod "ba64c0ce-499f-4940-9060-7d1e4b187ad9" (UID: "ba64c0ce-499f-4940-9060-7d1e4b187ad9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 05 20:20:16.497020 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.497001    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba64c0ce-499f-4940-9060-7d1e4b187ad9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba64c0ce-499f-4940-9060-7d1e4b187ad9" (UID: "ba64c0ce-499f-4940-9060-7d1e4b187ad9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 05 20:20:16.497179 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.497161    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba64c0ce-499f-4940-9060-7d1e4b187ad9-kube-api-access-pvkxk" (OuterVolumeSpecName: "kube-api-access-pvkxk") pod "ba64c0ce-499f-4940-9060-7d1e4b187ad9" (UID: "ba64c0ce-499f-4940-9060-7d1e4b187ad9"). InnerVolumeSpecName "kube-api-access-pvkxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 05 20:20:16.596123 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.596082    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvkxk\" (UniqueName: \"kubernetes.io/projected/ba64c0ce-499f-4940-9060-7d1e4b187ad9-kube-api-access-pvkxk\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:20:16.596281 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.596144    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba64c0ce-499f-4940-9060-7d1e4b187ad9-tls-certs\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:20:16.596281 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.596155    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba64c0ce-499f-4940-9060-7d1e4b187ad9-dshm\") on node \"ip-10-0-142-208.ec2.internal\" DevicePath \"\""
Sep 05 20:20:16.734037 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.734005    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"]
Sep 05 20:20:16.737910 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:16.737885    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-07fb4624-kserve-558876d58c-rksv9"]
Sep 05 20:20:17.434097 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:17.434062    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba64c0ce-499f-4940-9060-7d1e4b187ad9" path="/var/lib/kubelet/pods/ba64c0ce-499f-4940-9060-7d1e4b187ad9/volumes"
Sep 05 20:20:17.434514 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:17.434498    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2faded2-d1d3-405a-8de5-377ee528041d" path="/var/lib/kubelet/pods/e2faded2-d1d3-405a-8de5-377ee528041d/volumes"
Sep 05 20:20:47.783612 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:47.783534    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f67c45996-4jg96_720e095e-4556-4efc-8b0c-b1b68b037afa/authorino/0.log"
Sep 05 20:20:52.765217 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:52.765183    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-f5chv_0efde6da-00a5-46a8-a034-eb18da89c10a/manager/0.log"
Sep 05 20:20:52.874655 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:52.874616    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-627s6_31640114-9c04-4863-bd1f-da72c3101a2f/manager/0.log"
Sep 05 20:20:53.344658 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:53.344625    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-588fc9dcd8-699tm_4e640a9e-cca6-4e81-8be3-ccfd4b2845fd/copy-manifests/0.log"
Sep 05 20:20:53.350507 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:53.350480    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-588fc9dcd8-699tm_4e640a9e-cca6-4e81-8be3-ccfd4b2845fd/manager/0.log"
Sep 05 20:20:54.143073 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:54.143041    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29810640-dgxzx_bb5265ef-6e12-4e12-9df5-1f9ab3604f79/cleanup/0.log"
Sep 05 20:20:54.262519 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:54.262490    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29810655-xwprn_ce4899f7-bc8c-4015-bcdb-f6603a4a7f7f/cleanup/0.log"
Sep 05 20:20:55.108524 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:55.108492    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f67c45996-4jg96_720e095e-4556-4efc-8b0c-b1b68b037afa/authorino/0.log"
Sep 05 20:20:55.220938 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:55.220904    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-hjl4w_351f07d9-27e1-436e-81fc-f10f78cf16d0/manager/0.log"
Sep 05 20:20:55.323337 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:55.323309    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-vpk9d_f50c5f88-50ad-453a-b40e-593cff2208c4/manager/0.log"
Sep 05 20:20:55.428084 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:55.428005    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-98s8s_aa515af9-3fdb-403d-bd09-837323c156f6/kuadrant-console-plugin/0.log"
Sep 05 20:20:55.542262 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:55.542217    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-6pzml_ec15b1dd-1ce4-4134-a56f-49e8b8c84ca9/manager/0.log"
Sep 05 20:20:55.766983 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:55.766900    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-xfb64_02a54913-6879-42b0-96a4-e1584b934e40/manager/0.log"
Sep 05 20:20:56.094666 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:56.094641    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2_bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38/istio-proxy/0.log"
Sep 05 20:20:56.311878 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:56.311851    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84fbbb6876-94b2x_26be44b8-815a-4081-b02b-6efbb21a0dca/kube-auth-proxy/0.log"
Sep 05 20:20:56.521905 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:56.521827    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h7l4s_0d56147e-8a67-48db-8780-0c686acf83da/istio-proxy/1.log"
Sep 05 20:20:56.632567 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:56.632540    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-75cn5_bbb5f281-7a4f-4568-adf6-945df3d1e160/payload-pre-processing/0.log"
Sep 05 20:20:56.855747 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:56.855716    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-54d767fc5d-pv29k_a9c02cc0-cabd-4594-b23b-da027765f68f/router/0.log"
Sep 05 20:20:57.177614 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:20:57.177541    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-nnhhh_ff2eb6d6-81b7-4c8a-8dcf-6e40c3fd1a5c/main/0.log"
Sep 05 20:21:03.184797 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:03.184760    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hdj6r/must-gather-66vwh"]
Sep 05 20:21:03.185301 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:03.185276    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba64c0ce-499f-4940-9060-7d1e4b187ad9" containerName="main"
Sep 05 20:21:03.185301 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:03.185302    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba64c0ce-499f-4940-9060-7d1e4b187ad9" containerName="main"
Sep 05 20:21:03.185457 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:03.185338    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2faded2-d1d3-405a-8de5-377ee528041d" containerName="main"
Sep 05 20:21:03.185457 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:03.185350    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2faded2-d1d3-405a-8de5-377ee528041d" containerName="main"
Sep 05 20:21:03.185556 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:03.185481    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba64c0ce-499f-4940-9060-7d1e4b187ad9" containerName="main"
Sep 05 20:21:03.185556 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:03.185537    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2faded2-d1d3-405a-8de5-377ee528041d" containerName="main"
Sep 05 20:21:03.188960 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:03.188940    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hdj6r/must-gather-66vwh"
Sep 05 20:21:03.191076 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:03.191053    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hdj6r\"/\"kube-root-ca.crt\""
Sep 05 20:21:03.191202 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:03.191076    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hdj6r\"/\"openshift-service-ca.crt\""
Sep 05 20:21:03.191525 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:03.191491    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hdj6r\"/\"default-dockercfg-bdppc\""
Sep 05 20:21:03.196284 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:03.196254    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hdj6r/must-gather-66vwh"]
Sep 05 20:21:03.211385 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:03.211351    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm5wj\" (UniqueName: \"kubernetes.io/projected/11f47338-bcdf-40af-8499-5151fef27af4-kube-api-access-zm5wj\") pod \"must-gather-66vwh\" (UID: \"11f47338-bcdf-40af-8499-5151fef27af4\") " pod="openshift-must-gather-hdj6r/must-gather-66vwh"
Sep 05 20:21:03.211528 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:03.211425    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/11f47338-bcdf-40af-8499-5151fef27af4-must-gather-output\") pod \"must-gather-66vwh\" (UID: \"11f47338-bcdf-40af-8499-5151fef27af4\") " pod="openshift-must-gather-hdj6r/must-gather-66vwh"
Sep 05 20:21:03.312068 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:03.312020    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/11f47338-bcdf-40af-8499-5151fef27af4-must-gather-output\") pod \"must-gather-66vwh\" (UID: \"11f47338-bcdf-40af-8499-5151fef27af4\") " pod="openshift-must-gather-hdj6r/must-gather-66vwh"
Sep 05 20:21:03.312325 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:03.312149    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm5wj\" (UniqueName: \"kubernetes.io/projected/11f47338-bcdf-40af-8499-5151fef27af4-kube-api-access-zm5wj\") pod \"must-gather-66vwh\" (UID: \"11f47338-bcdf-40af-8499-5151fef27af4\") " pod="openshift-must-gather-hdj6r/must-gather-66vwh"
Sep 05 20:21:03.312447 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:03.312426    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/11f47338-bcdf-40af-8499-5151fef27af4-must-gather-output\") pod \"must-gather-66vwh\" (UID: \"11f47338-bcdf-40af-8499-5151fef27af4\") " pod="openshift-must-gather-hdj6r/must-gather-66vwh"
Sep 05 20:21:03.322471 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:03.322442    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm5wj\" (UniqueName: \"kubernetes.io/projected/11f47338-bcdf-40af-8499-5151fef27af4-kube-api-access-zm5wj\") pod \"must-gather-66vwh\" (UID: \"11f47338-bcdf-40af-8499-5151fef27af4\") " pod="openshift-must-gather-hdj6r/must-gather-66vwh"
Sep 05 20:21:03.499876 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:03.499770    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hdj6r/must-gather-66vwh"
Sep 05 20:21:03.775065 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:03.775035    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hdj6r/must-gather-66vwh"]
Sep 05 20:21:03.776693 ip-10-0-142-208 kubenswrapper[2583]: W0905 20:21:03.776669    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11f47338_bcdf_40af_8499_5151fef27af4.slice/crio-ce656bcba9ea43350225e70f60823695171a81c4bcba01343c442d8b4b353bcf WatchSource:0}: Error finding container ce656bcba9ea43350225e70f60823695171a81c4bcba01343c442d8b4b353bcf: Status 404 returned error can't find the container with id ce656bcba9ea43350225e70f60823695171a81c4bcba01343c442d8b4b353bcf
Sep 05 20:21:03.778596 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:03.778576    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 05 20:21:04.683536 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:04.683468    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hdj6r/must-gather-66vwh" event={"ID":"11f47338-bcdf-40af-8499-5151fef27af4","Type":"ContainerStarted","Data":"ce656bcba9ea43350225e70f60823695171a81c4bcba01343c442d8b4b353bcf"}
Sep 05 20:21:05.694589 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:05.694543    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hdj6r/must-gather-66vwh" event={"ID":"11f47338-bcdf-40af-8499-5151fef27af4","Type":"ContainerStarted","Data":"b21fcf681927397c4bbfb3480408f5c182d931d94904015fb0c33b69d2bb36dd"}
Sep 05 20:21:05.695072 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:05.694615    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hdj6r/must-gather-66vwh" event={"ID":"11f47338-bcdf-40af-8499-5151fef27af4","Type":"ContainerStarted","Data":"bb409ac493d30c0dd5b12b9bf761ead5039587143e166b985e646202e7b319e4"}
Sep 05 20:21:05.710727 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:05.710650    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hdj6r/must-gather-66vwh" podStartSLOduration=1.834854161 podStartE2EDuration="2.710628489s" totalImagesPullingTime="875.774328ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 20:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:21:05.708427844 +0000 UTC m=+2264.738059750" watchObservedRunningTime="2026-09-05 20:21:05.710628489 +0000 UTC m=+2264.740260394"
Sep 05 20:21:06.258491 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:06.258453    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ccth2_83377d0d-0308-4613-bd53-04d2cb4adc3b/global-pull-secret-syncer/0.log"
Sep 05 20:21:06.385555 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:06.385520    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sdb8j_f4df1d96-03b4-4947-aa18-86dd66548441/konnectivity-agent/0.log"
Sep 05 20:21:06.466126 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:06.466072    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-208.ec2.internal_ef30841ed7c762f04a60906c21dc14a2/haproxy/0.log"
Sep 05 20:21:10.810544 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:10.810507    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f67c45996-4jg96_720e095e-4556-4efc-8b0c-b1b68b037afa/authorino/0.log"
Sep 05 20:21:10.843472 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:10.843038    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-hjl4w_351f07d9-27e1-436e-81fc-f10f78cf16d0/manager/0.log"
Sep 05 20:21:10.866771 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:10.866737    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-vpk9d_f50c5f88-50ad-453a-b40e-593cff2208c4/manager/0.log"
Sep 05 20:21:10.891588 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:10.891554    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-98s8s_aa515af9-3fdb-403d-bd09-837323c156f6/kuadrant-console-plugin/0.log"
Sep 05 20:21:10.937783 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:10.937692    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-6pzml_ec15b1dd-1ce4-4134-a56f-49e8b8c84ca9/manager/0.log"
Sep 05 20:21:11.038884 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:11.038848    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-xfb64_02a54913-6879-42b0-96a4-e1584b934e40/manager/0.log"
Sep 05 20:21:12.842048 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:12.842009    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-wm7bd_cd029360-95f2-45c6-9648-1fe5fe757822/cluster-monitoring-operator/0.log"
Sep 05 20:21:12.870715 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:12.870679    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-vptgc_4ed7a58e-752e-4a92-a48b-6db480b280d3/kube-state-metrics/0.log"
Sep 05 20:21:12.887407 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:12.887378    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-vptgc_4ed7a58e-752e-4a92-a48b-6db480b280d3/kube-rbac-proxy-main/0.log"
Sep 05 20:21:12.911936 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:12.911896    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-vptgc_4ed7a58e-752e-4a92-a48b-6db480b280d3/kube-rbac-proxy-self/0.log"
Sep 05 20:21:13.080182 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:13.080088    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-llgfz_c82b3d00-c6ca-4e58-a4dc-e96f708cec4c/node-exporter/0.log"
Sep 05 20:21:13.103037 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:13.102951    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-llgfz_c82b3d00-c6ca-4e58-a4dc-e96f708cec4c/kube-rbac-proxy/0.log"
Sep 05 20:21:13.126749 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:13.126717    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-llgfz_c82b3d00-c6ca-4e58-a4dc-e96f708cec4c/init-textfile/0.log"
Sep 05 20:21:13.230447 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:13.230412    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-wj2f7_1342e506-7bb6-4af1-82cc-c1e8139193e1/kube-rbac-proxy-main/0.log"
Sep 05 20:21:13.252658 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:13.252624    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-wj2f7_1342e506-7bb6-4af1-82cc-c1e8139193e1/kube-rbac-proxy-self/0.log"
Sep 05 20:21:13.274806 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:13.274778    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-wj2f7_1342e506-7bb6-4af1-82cc-c1e8139193e1/openshift-state-metrics/0.log"
Sep 05 20:21:17.189452 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:17.189416    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q"]
Sep 05 20:21:17.195976 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:17.195946    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q"
Sep 05 20:21:17.201153 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:17.201054    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q"]
Sep 05 20:21:17.274641 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:17.274604    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/33ca7902-f1f8-4e28-8a70-2ae31753aefe-proc\") pod \"perf-node-gather-daemonset-l4s7q\" (UID: \"33ca7902-f1f8-4e28-8a70-2ae31753aefe\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q"
Sep 05 20:21:17.274841 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:17.274662    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/33ca7902-f1f8-4e28-8a70-2ae31753aefe-podres\") pod \"perf-node-gather-daemonset-l4s7q\" (UID: \"33ca7902-f1f8-4e28-8a70-2ae31753aefe\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q"
Sep 05 20:21:17.274841 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:17.274725    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33ca7902-f1f8-4e28-8a70-2ae31753aefe-sys\") pod \"perf-node-gather-daemonset-l4s7q\" (UID: \"33ca7902-f1f8-4e28-8a70-2ae31753aefe\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q"
Sep 05 20:21:17.274841 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:17.274755    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khwxz\" (UniqueName: \"kubernetes.io/projected/33ca7902-f1f8-4e28-8a70-2ae31753aefe-kube-api-access-khwxz\") pod \"perf-node-gather-daemonset-l4s7q\" (UID: \"33ca7902-f1f8-4e28-8a70-2ae31753aefe\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q"
Sep 05 20:21:17.275028 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:17.274859    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33ca7902-f1f8-4e28-8a70-2ae31753aefe-lib-modules\") pod \"perf-node-gather-daemonset-l4s7q\" (UID: \"33ca7902-f1f8-4e28-8a70-2ae31753aefe\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q"
Sep 05 20:21:17.286954 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:17.286924    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-ktdpg_4876fe64-e8bb-4e38-9076-4aa7eaa057d5/volume-data-source-validator/0.log"
Sep 05 20:21:17.376253 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:17.376212    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/33ca7902-f1f8-4e28-8a70-2ae31753aefe-proc\") pod \"perf-node-gather-daemonset-l4s7q\" (UID: \"33ca7902-f1f8-4e28-8a70-2ae31753aefe\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q"
Sep 05 20:21:17.376542 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:17.376510    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/33ca7902-f1f8-4e28-8a70-2ae31753aefe-podres\") pod \"perf-node-gather-daemonset-l4s7q\" (UID: \"33ca7902-f1f8-4e28-8a70-2ae31753aefe\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q"
Sep 05 20:21:17.376706 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:17.376685    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33ca7902-f1f8-4e28-8a70-2ae31753aefe-sys\") pod \"perf-node-gather-daemonset-l4s7q\" (UID: \"33ca7902-f1f8-4e28-8a70-2ae31753aefe\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q"
Sep 05 20:21:17.376822 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:17.376808    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khwxz\" (UniqueName: \"kubernetes.io/projected/33ca7902-f1f8-4e28-8a70-2ae31753aefe-kube-api-access-khwxz\") pod \"perf-node-gather-daemonset-l4s7q\" (UID: \"33ca7902-f1f8-4e28-8a70-2ae31753aefe\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q"
Sep 05 20:21:17.376960 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:17.376946    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33ca7902-f1f8-4e28-8a70-2ae31753aefe-lib-modules\") pod \"perf-node-gather-daemonset-l4s7q\" (UID: \"33ca7902-f1f8-4e28-8a70-2ae31753aefe\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q"
Sep 05 20:21:17.377273 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:17.377255    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33ca7902-f1f8-4e28-8a70-2ae31753aefe-lib-modules\") pod \"perf-node-gather-daemonset-l4s7q\" (UID: \"33ca7902-f1f8-4e28-8a70-2ae31753aefe\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q"
Sep 05 20:21:17.377465 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:17.377442    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/33ca7902-f1f8-4e28-8a70-2ae31753aefe-proc\") pod \"perf-node-gather-daemonset-l4s7q\" (UID: \"33ca7902-f1f8-4e28-8a70-2ae31753aefe\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q"
Sep 05 20:21:17.377644 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:17.377632    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/33ca7902-f1f8-4e28-8a70-2ae31753aefe-podres\") pod \"perf-node-gather-daemonset-l4s7q\" (UID: \"33ca7902-f1f8-4e28-8a70-2ae31753aefe\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q"
Sep 05 20:21:17.377772 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:17.377760    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33ca7902-f1f8-4e28-8a70-2ae31753aefe-sys\") pod \"perf-node-gather-daemonset-l4s7q\" (UID: \"33ca7902-f1f8-4e28-8a70-2ae31753aefe\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q"
Sep 05 20:21:17.388573 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:17.388547    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khwxz\" (UniqueName: \"kubernetes.io/projected/33ca7902-f1f8-4e28-8a70-2ae31753aefe-kube-api-access-khwxz\") pod \"perf-node-gather-daemonset-l4s7q\" (UID: \"33ca7902-f1f8-4e28-8a70-2ae31753aefe\") " pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q"
Sep 05 20:21:17.513414 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:17.513319    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q"
Sep 05 20:21:17.802283 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:17.802251    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q"]
Sep 05 20:21:18.644398 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:18.644360    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zj8hk_b1df5064-2ffa-4312-b6f3-cd6dd02c156a/dns/0.log"
Sep 05 20:21:18.667188 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:18.667150    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zj8hk_b1df5064-2ffa-4312-b6f3-cd6dd02c156a/kube-rbac-proxy/0.log"
Sep 05 20:21:18.711651 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:18.711620    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5h45d_0216ba9d-7122-47a3-86eb-8d14ddb478b0/dns-node-resolver/0.log"
Sep 05 20:21:18.810211 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:18.810166    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q" event={"ID":"33ca7902-f1f8-4e28-8a70-2ae31753aefe","Type":"ContainerStarted","Data":"d184526cf7b1dfad5a33fd19499e4fd10802fe660a9ee65fd77241295f335983"}
Sep 05 20:21:18.810409 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:18.810245    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q" event={"ID":"33ca7902-f1f8-4e28-8a70-2ae31753aefe","Type":"ContainerStarted","Data":"ca59c6a860147c13d195cfcd6e674a767fd2cd3dcd41daf10fa12bfb563dd45e"}
Sep 05 20:21:18.810506 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:18.810478    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q"
Sep 05 20:21:18.825585 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:18.825521    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q" podStartSLOduration=1.825500834 podStartE2EDuration="1.825500834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 20:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 20:21:18.824459905 +0000 UTC m=+2277.854091811" watchObservedRunningTime="2026-09-05 20:21:18.825500834 +0000 UTC m=+2277.855132740"
Sep 05 20:21:19.260720 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:19.260688    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-fd756675f-mnwfn_c26eb860-b9c1-40d3-a112-44e5fca931b0/registry/0.log"
Sep 05 20:21:19.336251 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:19.336215    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z6kd9_02b1f61a-c064-4c08-9bd4-cbe7fe25c934/node-ca/0.log"
Sep 05 20:21:20.242294 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:20.242258    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2_bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38/istio-proxy/0.log"
Sep 05 20:21:20.778726 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:20.778693    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84fbbb6876-94b2x_26be44b8-815a-4081-b02b-6efbb21a0dca/kube-auth-proxy/0.log"
Sep 05 20:21:20.832373 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:20.832336    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h7l4s_0d56147e-8a67-48db-8780-0c686acf83da/istio-proxy/0.log"
Sep 05 20:21:20.845343 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:20.845313    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h7l4s_0d56147e-8a67-48db-8780-0c686acf83da/istio-proxy/1.log"
Sep 05 20:21:20.883502 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:20.883469    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-75cn5_bbb5f281-7a4f-4568-adf6-945df3d1e160/payload-pre-processing/0.log"
Sep 05 20:21:20.951568 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:20.951535    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-54d767fc5d-pv29k_a9c02cc0-cabd-4594-b23b-da027765f68f/router/0.log"
Sep 05 20:21:21.801459 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:21.801419    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6ls6n_b3f8d9ae-aa9e-4eaa-ae8e-97526bc3add9/serve-healthcheck-canary/0.log"
Sep 05 20:21:22.466704 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:22.466676    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pgxbk_7a9b2788-2ab7-4c18-b80d-b5192c2909e0/kube-rbac-proxy/0.log"
Sep 05 20:21:22.486678 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:22.486647    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pgxbk_7a9b2788-2ab7-4c18-b80d-b5192c2909e0/exporter/0.log"
Sep 05 20:21:22.507051 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:22.506994    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pgxbk_7a9b2788-2ab7-4c18-b80d-b5192c2909e0/extractor/0.log"
Sep 05 20:21:24.548594 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:24.548560    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-f5chv_0efde6da-00a5-46a8-a034-eb18da89c10a/manager/0.log"
Sep 05 20:21:24.768555 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:24.768522    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-627s6_31640114-9c04-4863-bd1f-da72c3101a2f/manager/0.log"
Sep 05 20:21:24.829725 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:24.829695    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hdj6r/perf-node-gather-daemonset-l4s7q"
Sep 05 20:21:25.021216 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:25.021185    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-588fc9dcd8-699tm_4e640a9e-cca6-4e81-8be3-ccfd4b2845fd/manager/0.log"
Sep 05 20:21:25.041053 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:25.041025    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-588fc9dcd8-699tm_4e640a9e-cca6-4e81-8be3-ccfd4b2845fd/copy-manifests/0.log"
Sep 05 20:21:27.449389 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:27.449297    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2_bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38/istio-proxy/0.log"
Sep 05 20:21:27.987867 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:27.987813    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84fbbb6876-94b2x_26be44b8-815a-4081-b02b-6efbb21a0dca/kube-auth-proxy/0.log"
Sep 05 20:21:28.023496 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:28.023458    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h7l4s_0d56147e-8a67-48db-8780-0c686acf83da/istio-proxy/0.log"
Sep 05 20:21:28.038296 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:28.038266    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h7l4s_0d56147e-8a67-48db-8780-0c686acf83da/istio-proxy/1.log"
Sep 05 20:21:28.073579 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:28.073506    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-75cn5_bbb5f281-7a4f-4568-adf6-945df3d1e160/payload-pre-processing/0.log"
Sep 05 20:21:28.136541 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:28.136512    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-54d767fc5d-pv29k_a9c02cc0-cabd-4594-b23b-da027765f68f/router/0.log"
Sep 05 20:21:29.745769 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:29.745734    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-8gdm2_bc0e00bd-9a7e-45a5-93d7-2fd24fad5e38/istio-proxy/0.log"
Sep 05 20:21:30.287581 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:30.287535    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84fbbb6876-94b2x_26be44b8-815a-4081-b02b-6efbb21a0dca/kube-auth-proxy/0.log"
Sep 05 20:21:30.321262 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:30.321214    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h7l4s_0d56147e-8a67-48db-8780-0c686acf83da/istio-proxy/0.log"
Sep 05 20:21:30.332143 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:30.332104    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h7l4s_0d56147e-8a67-48db-8780-0c686acf83da/istio-proxy/1.log"
Sep 05 20:21:30.361072 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:30.361039    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-75cn5_bbb5f281-7a4f-4568-adf6-945df3d1e160/payload-pre-processing/0.log"
Sep 05 20:21:30.422521 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:30.422491    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-54d767fc5d-pv29k_a9c02cc0-cabd-4594-b23b-da027765f68f/router/0.log"
Sep 05 20:21:32.310093 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:32.310063    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-4v2k6_5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88/kube-storage-version-migrator-operator/1.log"
Sep 05 20:21:32.310980 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:32.310940    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-4v2k6_5c2c0c10-9d8c-4ceb-adb3-a963ef9a9a88/kube-storage-version-migrator-operator/0.log"
Sep 05 20:21:33.609838 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:33.609809    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wzs5q_334781cf-bbb8-46db-a50c-4d13f6e9f204/kube-multus-additional-cni-plugins/0.log"
Sep 05 20:21:33.633554 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:33.633528    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wzs5q_334781cf-bbb8-46db-a50c-4d13f6e9f204/egress-router-binary-copy/0.log"
Sep 05 20:21:33.654065 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:33.654031    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wzs5q_334781cf-bbb8-46db-a50c-4d13f6e9f204/cni-plugins/0.log"
Sep 05 20:21:33.674168 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:33.674141    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wzs5q_334781cf-bbb8-46db-a50c-4d13f6e9f204/bond-cni-plugin/0.log"
Sep 05 20:21:33.694992 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:33.694964    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wzs5q_334781cf-bbb8-46db-a50c-4d13f6e9f204/routeoverride-cni/0.log"
Sep 05 20:21:33.716717 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:33.716695    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wzs5q_334781cf-bbb8-46db-a50c-4d13f6e9f204/whereabouts-cni-bincopy/0.log"
Sep 05 20:21:33.737036 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:33.737012    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wzs5q_334781cf-bbb8-46db-a50c-4d13f6e9f204/whereabouts-cni/0.log"
Sep 05 20:21:33.792517 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:33.792488    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-grttc_2809e3e1-0bd8-4cc6-baa9-96ebbb7486f3/kube-multus/0.log"
Sep 05 20:21:33.996004 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:33.995893    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rwwbl_b9720241-ac7b-4d79-9644-233caff66a8c/network-metrics-daemon/0.log"
Sep 05 20:21:34.011619 ip-10-0-142-208 kubenswrapper[2583]: I0905 20:21:34.011597    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rwwbl_b9720241-ac7b-4d79-9644-233caff66a8c/kube-rbac-proxy/0.log"