Sep 18 13:13:01.437470 ip-10-0-129-234 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 18 13:13:01.437484 ip-10-0-129-234 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 18 13:13:01.437494 ip-10-0-129-234 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 18 13:13:01.437709 ip-10-0-129-234 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 18 13:13:11.632001 ip-10-0-129-234 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 18 13:13:11.632015 ip-10-0-129-234 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 83b4c79dcbfd4ac285dd71ef008843c7 --
Sep 18 13:14:51.430908 ip-10-0-129-234 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 13:14:51.850655 ip-10-0-129-234 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 13:14:51.850655 ip-10-0-129-234 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 13:14:51.850655 ip-10-0-129-234 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 13:14:51.850655 ip-10-0-129-234 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 13:14:51.850655 ip-10-0-129-234 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 13:14:51.853680 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.853580    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 13:14:51.860259 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860244    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:14:51.860259 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860260    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:14:51.860327 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860264    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:14:51.860327 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860267    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:14:51.860327 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860270    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:14:51.860327 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860274    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:14:51.860327 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860277    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:14:51.860327 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860280    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:14:51.860327 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860283    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:14:51.860327 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860286    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:14:51.860327 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860289    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:14:51.860327 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860292    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:14:51.860327 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860294    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:14:51.860327 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860297    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:14:51.860327 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860299    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:14:51.860327 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860303    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:14:51.860327 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860305    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:14:51.860327 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860308    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:14:51.860327 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860312    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:14:51.860327 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860314    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:14:51.860327 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860317    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:14:51.860327 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860319    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:14:51.860829 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860322    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:14:51.860829 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860324    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:14:51.860829 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860327    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:14:51.860829 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860330    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:14:51.860829 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860332    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:14:51.860829 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860335    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:14:51.860829 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860338    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:14:51.860829 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860341    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:14:51.860829 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860344    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:14:51.860829 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860346    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:14:51.860829 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860349    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:14:51.860829 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860352    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:14:51.860829 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860354    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:14:51.860829 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860356    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:14:51.860829 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860359    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:14:51.860829 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860361    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:14:51.860829 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860364    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:14:51.860829 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860366    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:14:51.860829 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860369    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:14:51.860829 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860371    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:14:51.861367 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860374    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:14:51.861367 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860376    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:14:51.861367 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860378    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:14:51.861367 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860381    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:14:51.861367 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860385    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:14:51.861367 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860389    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:14:51.861367 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860392    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:14:51.861367 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860394    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:14:51.861367 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860397    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:14:51.861367 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860400    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:14:51.861367 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860402    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:14:51.861367 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860405    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:14:51.861367 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860407    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:14:51.861367 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860412    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:14:51.861367 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860415    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:14:51.861367 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860418    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:14:51.861367 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860421    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:14:51.861367 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860424    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:14:51.861367 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860428    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:14:51.861866 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860431    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:14:51.861866 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860433    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:14:51.861866 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860436    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:14:51.861866 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860439    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:14:51.861866 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860442    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:14:51.861866 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860446    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:14:51.861866 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860449    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:14:51.861866 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860452    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:14:51.861866 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860455    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:14:51.861866 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860458    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:14:51.861866 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860460    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:14:51.861866 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860463    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:14:51.861866 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860466    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:14:51.861866 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860469    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:14:51.861866 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860472    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:14:51.861866 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860474    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:14:51.861866 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860477    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:14:51.861866 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860480    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:14:51.861866 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860483    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:14:51.861866 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860486    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:14:51.862440 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860488    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:14:51.862440 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860491    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:14:51.862440 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860493    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:14:51.862440 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860496    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:14:51.862440 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860499    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:14:51.862440 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860501    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:14:51.862440 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860505    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:14:51.862440 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860507    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:14:51.862440 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860510    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:14:51.862440 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860513    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:14:51.862440 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860515    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:14:51.862440 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860518    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:14:51.862440 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860520    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:14:51.862440 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860523    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:14:51.862440 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860525    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:14:51.862440 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860528    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:14:51.862440 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860530    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:14:51.862440 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.860533    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:14:51.863088 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863075    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:14:51.863088 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863087    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:14:51.863144 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863091    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:14:51.863144 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863094    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:14:51.863144 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863098    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:14:51.863144 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863101    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:14:51.863144 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863103    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:14:51.863144 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863106    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:14:51.863144 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863109    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:14:51.863144 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863112    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:14:51.863144 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863114    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:14:51.863144 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863117    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:14:51.863144 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863119    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:14:51.863144 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863122    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:14:51.863144 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863125    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:14:51.863144 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863127    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:14:51.863144 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863130    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:14:51.863144 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863133    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:14:51.863144 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863136    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:14:51.863144 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863139    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:14:51.863144 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863141    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:14:51.863144 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863144    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:14:51.863144 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863147    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:14:51.863684 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863149    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:14:51.863684 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863152    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:14:51.863684 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863155    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:14:51.863684 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863158    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:14:51.863684 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863160    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:14:51.863684 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863163    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:14:51.863684 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863165    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:14:51.863684 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863168    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:14:51.863684 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863170    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:14:51.863684 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863173    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:14:51.863684 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863175    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:14:51.863684 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863178    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:14:51.863684 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863180    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:14:51.863684 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863183    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:14:51.863684 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863186    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:14:51.863684 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863188    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:14:51.863684 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863191    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:14:51.863684 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863193    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:14:51.863684 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863196    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:14:51.864146 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863198    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:14:51.864146 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863202    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:14:51.864146 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863206    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:14:51.864146 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863208    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:14:51.864146 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863211    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:14:51.864146 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863214    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:14:51.864146 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863217    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:14:51.864146 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863219    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:14:51.864146 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863222    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:14:51.864146 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863225    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:14:51.864146 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863228    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:14:51.864146 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863231    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:14:51.864146 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863234    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:14:51.864146 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863237    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:14:51.864146 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863242    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:14:51.864146 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863245    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:14:51.864146 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863248    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:14:51.864146 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863250    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:14:51.864146 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863253    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:14:51.864632 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863256    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:14:51.864632 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863258    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:14:51.864632 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863261    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:14:51.864632 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863264    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:14:51.864632 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863268    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:14:51.864632 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863271    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:14:51.864632 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863276    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:14:51.864632 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863279    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:14:51.864632 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863282    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:14:51.864632 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863285    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:14:51.864632 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863288    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:14:51.864632 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863291    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:14:51.864632 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863293    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:14:51.864632 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863296    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:14:51.864632 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863299    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:14:51.864632 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863301    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:14:51.864632 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863304    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:14:51.864632 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863306    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:14:51.865085 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863309    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:14:51.865085 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863311    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:14:51.865085 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863314    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:14:51.865085 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863317    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:14:51.865085 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863319    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:14:51.865085 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863323    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:14:51.865085 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863326    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:14:51.865085 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863329    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:14:51.865085 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863331    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:14:51.865085 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863334    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:14:51.865085 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863337    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:14:51.865085 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863339    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:14:51.865085 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863342    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:14:51.865085 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863344    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:14:51.865085 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863347    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:14:51.865085 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863349    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:14:51.865085 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863352    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:14:51.865085 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863354    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:14:51.865085 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863357    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:14:51.865085 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863360    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:14:51.865604 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863747    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:14:51.865604 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863753    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:14:51.865604 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863756    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:14:51.865604 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863759    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:14:51.865604 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863763    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:14:51.865604 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863767    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:14:51.865604 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863771    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:14:51.865604 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863774    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:14:51.865604 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863776    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:14:51.865604 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863779    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:14:51.865604 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863782    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:14:51.865604 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863784    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:14:51.865604 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863788    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:14:51.865604 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863791    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:14:51.865604 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863794    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:14:51.865604 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863797    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:14:51.865604 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863799    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:14:51.865604 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863802    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:14:51.865604 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863805    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:14:51.865604 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863808    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:14:51.866095 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863810    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:14:51.866095 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863814    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:14:51.866095 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863816    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:14:51.866095 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863819    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:14:51.866095 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863821    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:14:51.866095 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863824    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:14:51.866095 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863826    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:14:51.866095 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863829    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:14:51.866095 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863832    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:14:51.866095 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863834    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:14:51.866095 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863837    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:14:51.866095 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863839    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:14:51.866095 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863842    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:14:51.866095 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863844    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:14:51.866095 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863847    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:14:51.866095 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863849    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:14:51.866095 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863852    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:14:51.866095 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863855    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:14:51.866095 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863857    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:14:51.866095 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863860    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:14:51.866583 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863863    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:14:51.866583 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863865    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:14:51.866583 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863868    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:14:51.866583 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863870    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:14:51.866583 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863874    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:14:51.866583 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863876    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:14:51.866583 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863879    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:14:51.866583 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863882    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:14:51.866583 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863884    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:14:51.866583 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863888    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:14:51.866583 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863892    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:14:51.866583 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863895    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:14:51.866583 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863898    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:14:51.866583 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863901    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:14:51.866583 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863903    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:14:51.866583 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863906    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:14:51.866583 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863909    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:14:51.866583 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863911    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:14:51.866583 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863914    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:14:51.866583 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863917    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:14:51.867116 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863919    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:14:51.867116 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863922    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:14:51.867116 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863924    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:14:51.867116 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863927    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:14:51.867116 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863929    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:14:51.867116 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863932    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:14:51.867116 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863934    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:14:51.867116 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863937    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:14:51.867116 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863939    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:14:51.867116 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863942    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:14:51.867116 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863945    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:14:51.867116 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863947    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:14:51.867116 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863950    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:14:51.867116 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863954    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:14:51.867116 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863957    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:14:51.867116 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863960    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:14:51.867116 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863963    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:14:51.867116 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863966    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:14:51.867116 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863968    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:14:51.867589 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863971    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:14:51.867589 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863974    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:14:51.867589 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863976    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:14:51.867589 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863979    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:14:51.867589 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863982    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:14:51.867589 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863984    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:14:51.867589 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863987    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:14:51.867589 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863990    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:14:51.867589 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863992    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:14:51.867589 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863995    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:14:51.867589 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.863997    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:14:51.867589 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864000    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:14:51.867589 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864002    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:14:51.867589 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864005    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:14:51.867589 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864007    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:14:51.867589 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864010    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:14:51.867589 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864013    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:14:51.867589 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864015    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:14:51.867589 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864017    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:14:51.867589 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864020    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:14:51.868081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864083    2566 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 13:14:51.868081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864123    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 13:14:51.868081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864131    2566 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 13:14:51.868081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864136    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 13:14:51.868081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864140    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 13:14:51.868081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864143    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 13:14:51.868081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864148    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 13:14:51.868081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864152    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 13:14:51.868081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864155    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 13:14:51.868081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864159    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 13:14:51.868081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864162    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 13:14:51.868081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864166    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 13:14:51.868081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864169    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 13:14:51.868081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864172    2566 flags.go:64] FLAG: --cgroup-root=""
Sep 18 13:14:51.868081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864175    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 13:14:51.868081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864178    2566 flags.go:64] FLAG: --client-ca-file=""
Sep 18 13:14:51.868081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864181    2566 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 13:14:51.868081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864208    2566 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 13:14:51.868081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864212    2566 flags.go:64] FLAG: --cluster-domain=""
Sep 18 13:14:51.868081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864215    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 13:14:51.868081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864218    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 13:14:51.868081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864222    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 13:14:51.868081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864225    2566 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 13:14:51.868081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864229    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864232    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864235    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864238    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864241    2566 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864244    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864248    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864251    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864254    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864258    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864261    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864264    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864267    2566 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864270    2566 flags.go:64] FLAG: --enable-server="true"
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864274    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864279    2566 flags.go:64] FLAG: --event-burst="100"
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864282    2566 flags.go:64] FLAG: --event-qps="50"
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864285    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864288    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864291    2566 flags.go:64] FLAG: --eviction-hard=""
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864295    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864298    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864301    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864303    2566 flags.go:64] FLAG: --eviction-soft=""
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864306    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 13:14:51.868719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864309    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 13:14:51.869339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864312    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 13:14:51.869339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864316    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 13:14:51.869339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864319    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 13:14:51.869339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864322    2566 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 13:14:51.869339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864325    2566 flags.go:64] FLAG: --feature-gates=""
Sep 18 13:14:51.869339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864329    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 13:14:51.869339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864332    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 13:14:51.869339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864335    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 13:14:51.869339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864338    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 13:14:51.869339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864341    2566 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 13:14:51.869339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864344    2566 flags.go:64] FLAG: --help="false"
Sep 18 13:14:51.869339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864347    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-129-234.ec2.internal"
Sep 18 13:14:51.869339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864349    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 13:14:51.869339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864352    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 13:14:51.869339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864355    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 13:14:51.869339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864358    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 13:14:51.869339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864362    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 13:14:51.869339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864366    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 13:14:51.869339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864369    2566 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 13:14:51.869339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864372    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 13:14:51.869339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864375    2566 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 13:14:51.869339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864378    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 13:14:51.869339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864381    2566 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 13:14:51.869339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864384    2566 flags.go:64] FLAG: --kube-reserved=""
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864389    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864392    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864395    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864398    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864401    2566 flags.go:64] FLAG: --lock-file=""
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864403    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864406    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864409    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864414    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864417    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864421    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864424    2566 flags.go:64] FLAG: --logging-format="text"
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864427    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864430    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864433    2566 flags.go:64] FLAG: --manifest-url=""
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864436    2566 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864440    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864443    2566 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864447    2566 flags.go:64] FLAG: --max-pods="110"
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864450    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864453    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864456    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864459    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864462    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 13:14:51.869948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864465    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864469    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864476    2566 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864480    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864483    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864486    2566 flags.go:64] FLAG: --pod-cidr=""
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864489    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864494    2566 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864497    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864501    2566 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864504    2566 flags.go:64] FLAG: --port="10250"
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864507    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864510    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05d308d4995614367"
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864513    2566 flags.go:64] FLAG: --qos-reserved=""
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864516    2566 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864519    2566 flags.go:64] FLAG: --register-node="true"
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864523    2566 flags.go:64] FLAG: --register-with-taints=""
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864526    2566 flags.go:64] FLAG: --registry-burst="10"
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864529    2566 flags.go:64] FLAG: --registry-qps="5"
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864533    2566 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864536    2566 flags.go:64] FLAG: --reserved-memory=""
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864540    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864543    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864546    2566 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864549    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 13:14:51.870561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864552    2566 flags.go:64] FLAG: --runonce="false"
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864555    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864558    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864561    2566 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864564    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864567    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864570    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864573    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864577    2566 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864580    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864583    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864585    2566 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864588    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864606    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864610    2566 flags.go:64] FLAG: --system-cgroups=""
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864613    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864619    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864622    2566 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864625    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864629    2566 flags.go:64] FLAG: --tls-min-version=""
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864632    2566 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864635    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864638    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864641    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864645    2566 flags.go:64] FLAG: --v="2"
Sep 18 13:14:51.871196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864649    2566 flags.go:64] FLAG: --version="false"
Sep 18 13:14:51.871907 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864652    2566 flags.go:64] FLAG: --vmodule=""
Sep 18 13:14:51.871907 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864657    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 13:14:51.871907 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.864660    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 13:14:51.871907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864772    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:14:51.871907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864776    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:14:51.871907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864779    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:14:51.871907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864782    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:14:51.871907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864785    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:14:51.871907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864788    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:14:51.871907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864791    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:14:51.871907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864794    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:14:51.871907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864797    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:14:51.871907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864799    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:14:51.871907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864802    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:14:51.871907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864806    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:14:51.871907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864809    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:14:51.871907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864812    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:14:51.871907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864814    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:14:51.871907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864817    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:14:51.871907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864820    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:14:51.872551 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864822    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:14:51.872551 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864825    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:14:51.872551 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864829    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:14:51.872551 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864832    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:14:51.872551 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864834    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:14:51.872551 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864838    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:14:51.872551 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864842    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:14:51.872551 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864845    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:14:51.872551 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864848    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:14:51.872551 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864851    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:14:51.872551 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864854    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:14:51.872551 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864856    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:14:51.872551 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864859    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:14:51.872551 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864862    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:14:51.872551 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864865    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:14:51.872551 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864867    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:14:51.872551 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864870    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:14:51.872551 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864873    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:14:51.872551 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864876    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:14:51.872551 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864878    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:14:51.873348 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864881    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:14:51.873348 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864883    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:14:51.873348 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864886    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:14:51.873348 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864889    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:14:51.873348 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864891    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:14:51.873348 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864894    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:14:51.873348 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864898    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:14:51.873348 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864901    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:14:51.873348 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864904    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:14:51.873348 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864906    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:14:51.873348 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864909    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:14:51.873348 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864911    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:14:51.873348 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864914    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:14:51.873348 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864916    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:14:51.873348 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864920    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:14:51.873348 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864923    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:14:51.873348 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864925    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:14:51.873348 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864928    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:14:51.873348 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864931    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:14:51.873846 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864933    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:14:51.873846 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864936    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:14:51.873846 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864938    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:14:51.873846 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864941    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:14:51.873846 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864943    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:14:51.873846 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864946    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:14:51.873846 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864948    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:14:51.873846 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864951    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:14:51.873846 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864954    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:14:51.873846 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864957    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:14:51.873846 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864959    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:14:51.873846 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864962    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:14:51.873846 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864964    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:14:51.873846 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864967    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:14:51.873846 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864969    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:14:51.873846 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864972    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:14:51.873846 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864974    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:14:51.873846 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864977    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:14:51.873846 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864980    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:14:51.873846 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864984    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:14:51.874336 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864986    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:14:51.874336 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864989    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:14:51.874336 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864993    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:14:51.874336 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864996    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:14:51.874336 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.864999    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:14:51.874336 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.865002    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:14:51.874336 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.865004    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:14:51.874336 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.865008    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:14:51.874336 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.865011    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:14:51.874336 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.865014    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:14:51.874336 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.865016    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:14:51.874336 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.865019    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:14:51.874336 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.865022    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:14:51.874336 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.865024    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:14:51.874336 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.865027    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:14:51.874336 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.865029    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:14:51.874336 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.865032    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:14:51.874336 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.865034    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:14:51.874336 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.865037    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:14:51.874822 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.865040    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:14:51.874822 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.865042    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:14:51.874822 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.865047    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:14:51.874822 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.865050    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:14:51.874822 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.865611    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 13:14:51.874822 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.874012    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 13:14:51.874822 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.874027    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 13:14:51.874822 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.874047    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 13:14:51.874822 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.874055    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 13:14:51.874822 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874110    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:14:51.874822 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874114    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:14:51.874822 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874118    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:14:51.874822 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874121    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:14:51.874822 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874124    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:14:51.875193 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874127    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:14:51.875193 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874130    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:14:51.875193 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874133    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:14:51.875193 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874137    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:14:51.875193 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874139    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:14:51.875193 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874142    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:14:51.875193 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874145    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:14:51.875193 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874147    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:14:51.875193 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874150    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:14:51.875193 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874153    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:14:51.875193 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874156    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:14:51.875193 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874158    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:14:51.875193 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874161    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:14:51.875193 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874163    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:14:51.875193 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874166    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:14:51.875193 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874168    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:14:51.875193 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874171    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:14:51.875193 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874174    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:14:51.875193 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874177    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:14:51.875672 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874181    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:14:51.875672 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874185    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:14:51.875672 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874188    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:14:51.875672 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874192    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:14:51.875672 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874195    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:14:51.875672 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874199    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:14:51.875672 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874202    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:14:51.875672 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874204    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:14:51.875672 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874207    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:14:51.875672 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874210    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:14:51.875672 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874212    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:14:51.875672 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874215    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:14:51.875672 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874218    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:14:51.875672 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874221    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:14:51.875672 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874223    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:14:51.875672 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874226    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:14:51.875672 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874228    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:14:51.875672 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874232    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:14:51.875672 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874236    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:14:51.876183 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874239    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:14:51.876183 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874242    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:14:51.876183 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874246    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:14:51.876183 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874259    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:14:51.876183 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874262    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:14:51.876183 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874265    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:14:51.876183 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874268    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:14:51.876183 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874271    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:14:51.876183 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874274    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:14:51.876183 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874277    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:14:51.876183 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874279    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:14:51.876183 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874282    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:14:51.876183 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874284    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:14:51.876183 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874287    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:14:51.876183 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874290    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:14:51.876183 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874293    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:14:51.876183 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874295    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:14:51.876183 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874298    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:14:51.876183 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874301    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:14:51.876695 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874304    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:14:51.876695 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874307    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:14:51.876695 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874310    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:14:51.876695 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874313    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:14:51.876695 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874315    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:14:51.876695 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874318    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:14:51.876695 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874321    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:14:51.876695 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874324    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:14:51.876695 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874326    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:14:51.876695 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874329    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:14:51.876695 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874331    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:14:51.876695 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874334    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:14:51.876695 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874337    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:14:51.876695 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874340    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:14:51.876695 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874342    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:14:51.876695 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874345    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:14:51.876695 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874348    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:14:51.876695 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874352    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:14:51.876695 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874354    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:14:51.876695 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874357    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:14:51.877167 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874360    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:14:51.877167 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874362    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:14:51.877167 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874365    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:14:51.877167 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874367    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:14:51.877167 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874370    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:14:51.877167 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874373    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:14:51.877167 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874375    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:14:51.877167 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874378    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:14:51.877167 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874381    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:14:51.877167 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874383    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:14:51.877167 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874386    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:14:51.877167 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874389    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:14:51.877167 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874392    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:14:51.877167 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874394    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:14:51.877167 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874397    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:14:51.877167 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874399    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:14:51.877167 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874402    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:14:51.877565 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.874407    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 13:14:51.877565 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874516    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:14:51.877565 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874520    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:14:51.877565 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874523    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:14:51.877565 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874526    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:14:51.877565 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874529    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:14:51.877565 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874532    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:14:51.877565 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874535    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:14:51.877565 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874538    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:14:51.877565 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874541    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:14:51.877565 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874543    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:14:51.877565 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874546    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:14:51.877565 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874549    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:14:51.877565 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874553    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:14:51.877565 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874557    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:14:51.877951 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874560    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:14:51.877951 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874563    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:14:51.877951 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874566    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:14:51.877951 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874569    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:14:51.877951 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874572    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:14:51.877951 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874574    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:14:51.877951 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874577    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:14:51.877951 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874579    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:14:51.877951 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874582    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:14:51.877951 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874585    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:14:51.877951 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874588    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:14:51.877951 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874602    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:14:51.877951 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874605    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:14:51.877951 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874608    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:14:51.877951 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874611    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:14:51.877951 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874613    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:14:51.877951 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874616    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:14:51.877951 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874619    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:14:51.877951 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874622    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:14:51.878408 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874625    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:14:51.878408 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874627    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:14:51.878408 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874630    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:14:51.878408 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874633    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:14:51.878408 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874635    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:14:51.878408 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874638    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:14:51.878408 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874640    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:14:51.878408 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874643    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:14:51.878408 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874646    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:14:51.878408 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874648    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:14:51.878408 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874651    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:14:51.878408 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874653    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:14:51.878408 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874661    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:14:51.878408 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874664    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:14:51.878408 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874667    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:14:51.878408 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874669    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:14:51.878408 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874672    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:14:51.878408 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874675    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:14:51.878408 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874678    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:14:51.878408 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874680    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:14:51.878907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874683    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:14:51.878907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874686    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:14:51.878907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874688    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:14:51.878907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874691    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:14:51.878907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874694    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:14:51.878907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874698    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:14:51.878907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874701    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:14:51.878907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874703    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:14:51.878907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874706    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:14:51.878907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874709    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:14:51.878907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874711    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:14:51.878907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874714    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:14:51.878907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874717    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:14:51.878907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874719    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:14:51.878907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874722    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:14:51.878907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874724    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:14:51.878907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874727    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:14:51.878907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874730    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:14:51.878907 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874732    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:14:51.879381 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874735    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:14:51.879381 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874737    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:14:51.879381 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874740    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:14:51.879381 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874743    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:14:51.879381 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874745    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:14:51.879381 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874748    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:14:51.879381 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874751    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:14:51.879381 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874754    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:14:51.879381 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874756    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:14:51.879381 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874759    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:14:51.879381 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874762    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:14:51.879381 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874764    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:14:51.879381 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874767    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:14:51.879381 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874769    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:14:51.879381 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874772    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:14:51.879381 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874774    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:14:51.879381 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874777    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:14:51.879381 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874781    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:14:51.879381 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874784    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:14:51.879381 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874787    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:14:51.879887 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874790    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:14:51.879887 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874793    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:14:51.879887 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874795    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:14:51.879887 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874798    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:14:51.879887 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874800    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:14:51.879887 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874803    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:14:51.879887 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:51.874806    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:14:51.879887 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.874810    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 13:14:51.879887 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.874910    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 13:14:51.879887 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.877116    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 13:14:51.879887 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.878102    2566 server.go:1019] "Starting client certificate rotation"
Sep 18 13:14:51.879887 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.878200    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 13:14:51.879887 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.878228    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 13:14:51.880844 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.880831    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 13:14:51.882126 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.882102    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 13:14:51.891358 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.891337    2566 log.go:25] "Validated CRI v1 runtime API"
Sep 18 13:14:51.892957 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.892941    2566 log.go:25] "Validated CRI v1 image API"
Sep 18 13:14:51.896041 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.896024    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 13:14:51.898309 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.898290    2566 fs.go:135] Filesystem UUIDs: map[58f57ac7-fb78-47cc-bfcb-99ee8ea5c824:/dev/nvme0n1p3 68b6c78e-1afc-495b-8d4c-ac5294baec3b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 18 13:14:51.898374 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.898309    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 13:14:51.903417 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.903313    2566 manager.go:216] Machine: {Timestamp:2026-09-18 13:14:51.901802009 +0000 UTC m=+0.336678404 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103452 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a1e2914deff036d3c230b82779e7c SystemUUID:ec2a1e29-14de-ff03-6d3c-230b82779e7c BootID:83b4c79d-cbfd-4ac2-85dd-71ef008843c7 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:df:a6:35:ad Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:df:a6:35:ad Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:bd:08:4e:51:d0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 13:14:51.903417 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.903415    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 13:14:51.903529 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.903491    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 13:14:51.906477 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.906451    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 13:14:51.906622 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.906478    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-234.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 13:14:51.907721 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.907711    2566 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 13:14:51.907755 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.907724    2566 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 13:14:51.907755 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.907731    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 13:14:51.908348 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.908339    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 13:14:51.908420 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.908410    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 13:14:51.912143 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.912131    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 13:14:51.912254 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.912245    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 13:14:51.912294 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.912273    2566 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 13:14:51.912294 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.912284    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 13:14:51.912374 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.912302    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 13:14:51.912374 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.912311    2566 kubelet.go:398] "Adding apiserver pod source"
Sep 18 13:14:51.912374 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.912320    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 13:14:51.913114 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.913092    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 13:14:51.913451 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.913434    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 13:14:51.913509 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.913458    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 13:14:51.916885 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.916867    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 13:14:51.919212 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.919198    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 13:14:51.919277 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.919229    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 13:14:51.920271 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.920260    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 13:14:51.920308 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.920275    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 13:14:51.920308 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.920281    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 13:14:51.920308 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.920287    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 13:14:51.920308 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.920297    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 13:14:51.920308 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.920303    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 13:14:51.920475 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.920321    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 13:14:51.920475 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.920327    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 13:14:51.920475 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.920334    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 13:14:51.920475 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.920339    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 13:14:51.920475 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.920348    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 13:14:51.920475 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.920353    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 13:14:51.921725 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.921714    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 13:14:51.921725 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.921724    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 13:14:51.924062 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.924043    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-234.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 13:14:51.924625 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:51.924575    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 13:14:51.924703 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:51.924661    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-234.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 13:14:51.925543 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.925530    2566 server.go:1268] "Started kubelet"
Sep 18 13:14:51.925617 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.925580    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 13:14:51.925727 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.925677    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 13:14:51.925772 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.925751    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 13:14:51.926364 ip-10-0-129-234 systemd[1]: Started Kubernetes Kubelet.
Sep 18 13:14:51.926912 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.926894    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 13:14:51.928278 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.928265    2566 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 13:14:51.930642 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:51.930625    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 13:14:51.932633 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.932619    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 13:14:51.933151 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.933132    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 13:14:51.933219 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:51.933197    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-234.ec2.internal\" not found"
Sep 18 13:14:51.933727 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.933709    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 13:14:51.933727 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.933712    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 13:14:51.933855 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.933741    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 13:14:51.933855 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.933789    2566 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 13:14:51.933855 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.933799    2566 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 13:14:51.934141 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.934028    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 13:14:51.934141 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.934141    2566 factory.go:55] Registering systemd factory
Sep 18 13:14:51.934212 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.934148    2566 factory.go:223] Registration of the systemd container factory successfully
Sep 18 13:14:51.934358 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.934347    2566 factory.go:153] Registering CRI-O factory
Sep 18 13:14:51.934358 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.934357    2566 factory.go:223] Registration of the crio container factory successfully
Sep 18 13:14:51.934435 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.934380    2566 factory.go:103] Registering Raw factory
Sep 18 13:14:51.934435 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.934390    2566 manager.go:1196] Started watching for new ooms in manager
Sep 18 13:14:51.934801 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.934787    2566 manager.go:349] Starting recovery of all containers
Sep 18 13:14:51.943653 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:51.943622    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-234.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 13:14:51.943776 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:51.943732    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 13:14:51.946610 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.946575    2566 manager.go:354] Recovery completed
Sep 18 13:14:51.947529 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:51.944000    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-234.ec2.internal.18d66c95a67d3fa3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-234.ec2.internal,UID:ip-10-0-129-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-234.ec2.internal,},FirstTimestamp:2026-09-18 13:14:51.925503907 +0000 UTC m=+0.360380302,LastTimestamp:2026-09-18 13:14:51.925503907 +0000 UTC m=+0.360380302,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-234.ec2.internal,}"
Sep 18 13:14:51.951454 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.951439    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:14:51.953146 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.953127    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:14:51.953219 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.953168    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:14:51.953219 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.953185    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:14:51.953769 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.953754    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 13:14:51.953816 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.953769    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 13:14:51.953816 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.953787    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 13:14:51.954486 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.954473    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:14:51.955269 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.955250    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:14:51.955349 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.955284    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:14:51.955349 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.955298    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:14:51.955772 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:51.955696    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-234.ec2.internal.18d66c95a82313f8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-234.ec2.internal,UID:ip-10-0-129-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-234.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-234.ec2.internal,},FirstTimestamp:2026-09-18 13:14:51.95314892 +0000 UTC m=+0.388025317,LastTimestamp:2026-09-18 13:14:51.95314892 +0000 UTC m=+0.388025317,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-234.ec2.internal,}"
Sep 18 13:14:51.955879 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.955837    2566 policy_none.go:49] "None policy: Start"
Sep 18 13:14:51.955879 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.955854    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 13:14:51.955879 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.955869    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 13:14:51.957866 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.957854    2566 policy_none.go:47] "Start"
Sep 18 13:14:51.959427 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.959410    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6sbpf"
Sep 18 13:14:51.967514 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:51.967427    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-234.ec2.internal.18d66c95a8237b6d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-234.ec2.internal,UID:ip-10-0-129-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-234.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-234.ec2.internal,},FirstTimestamp:2026-09-18 13:14:51.953175405 +0000 UTC m=+0.388051803,LastTimestamp:2026-09-18 13:14:51.953175405 +0000 UTC m=+0.388051803,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-234.ec2.internal,}"
Sep 18 13:14:51.977403 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.977383    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6sbpf"
Sep 18 13:14:51.980059 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:51.979983    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-234.ec2.internal.18d66c95a823bbc2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-234.ec2.internal,UID:ip-10-0-129-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-234.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-234.ec2.internal,},FirstTimestamp:2026-09-18 13:14:51.953191874 +0000 UTC m=+0.388068272,LastTimestamp:2026-09-18 13:14:51.953191874 +0000 UTC m=+0.388068272,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-234.ec2.internal,}"
Sep 18 13:14:51.983259 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.983247    2566 manager.go:337] "Starting Device Plugin manager"
Sep 18 13:14:51.983335 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:51.983290    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 13:14:51.983335 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.983303    2566 server.go:87] "Starting device plugin registration server"
Sep 18 13:14:51.983689 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.983676    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 13:14:51.983774 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.983691    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 13:14:51.984313 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.983861    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 13:14:51.984313 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.983929    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 13:14:51.984313 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:51.983936    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 13:14:52.015668 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:51.985299    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 13:14:52.015668 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:51.985353    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-234.ec2.internal\" not found"
Sep 18 13:14:52.068951 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.068910    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 13:14:52.070136 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.070120    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 13:14:52.070207 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.070172    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 13:14:52.070207 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.070198    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 13:14:52.070480 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:52.070240    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 13:14:52.073439 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.073419    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:14:52.084210 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.084187    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:14:52.085101 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.085086    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:14:52.085174 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.085116    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:14:52.085174 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.085127    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:14:52.085174 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.085150    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-234.ec2.internal"
Sep 18 13:14:52.093112 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.093088    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-234.ec2.internal"
Sep 18 13:14:52.093112 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:52.093113    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-234.ec2.internal\": node \"ip-10-0-129-234.ec2.internal\" not found"
Sep 18 13:14:52.147254 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:52.147164    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-234.ec2.internal\" not found"
Sep 18 13:14:52.170670 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.170650    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-234.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-234.ec2.internal"]
Sep 18 13:14:52.170762 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.170717    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:14:52.171671 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.171654    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:14:52.171773 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.171683    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:14:52.171773 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.171692    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:14:52.174012 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.173998    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:14:52.174138 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.174123    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-234.ec2.internal"
Sep 18 13:14:52.174196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.174159    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:14:52.174802 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.174787    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:14:52.174802 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.174800    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:14:52.174915 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.174811    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:14:52.174915 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.174820    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:14:52.174915 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.174820    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:14:52.174915 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.174841    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:14:52.176909 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.176895    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-234.ec2.internal"
Sep 18 13:14:52.176959 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.176918    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:14:52.177663 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.177642    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:14:52.177751 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.177672    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:14:52.177751 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.177688    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:14:52.201201 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:52.201184    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-234.ec2.internal\" not found" node="ip-10-0-129-234.ec2.internal"
Sep 18 13:14:52.205677 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:52.205662    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-234.ec2.internal\" not found" node="ip-10-0-129-234.ec2.internal"
Sep 18 13:14:52.235344 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.235321    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e90c85b650444ce894d74fab7d1e58d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-234.ec2.internal\" (UID: \"e90c85b650444ce894d74fab7d1e58d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-234.ec2.internal"
Sep 18 13:14:52.235423 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.235373    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e90c85b650444ce894d74fab7d1e58d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-234.ec2.internal\" (UID: \"e90c85b650444ce894d74fab7d1e58d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-234.ec2.internal"
Sep 18 13:14:52.235423 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.235411    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6137ead809380c4ef87cd83540755ba-config\") pod \"kube-apiserver-proxy-ip-10-0-129-234.ec2.internal\" (UID: \"d6137ead809380c4ef87cd83540755ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-234.ec2.internal"
Sep 18 13:14:52.247417 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:52.247399    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-234.ec2.internal\" not found"
Sep 18 13:14:52.335566 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.335527    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e90c85b650444ce894d74fab7d1e58d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-234.ec2.internal\" (UID: \"e90c85b650444ce894d74fab7d1e58d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-234.ec2.internal"
Sep 18 13:14:52.335566 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.335563    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6137ead809380c4ef87cd83540755ba-config\") pod \"kube-apiserver-proxy-ip-10-0-129-234.ec2.internal\" (UID: \"d6137ead809380c4ef87cd83540755ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-234.ec2.internal"
Sep 18 13:14:52.335770 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.335578    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e90c85b650444ce894d74fab7d1e58d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-234.ec2.internal\" (UID: \"e90c85b650444ce894d74fab7d1e58d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-234.ec2.internal"
Sep 18 13:14:52.335770 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.335638    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e90c85b650444ce894d74fab7d1e58d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-234.ec2.internal\" (UID: \"e90c85b650444ce894d74fab7d1e58d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-234.ec2.internal"
Sep 18 13:14:52.335770 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.335706    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e90c85b650444ce894d74fab7d1e58d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-234.ec2.internal\" (UID: \"e90c85b650444ce894d74fab7d1e58d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-234.ec2.internal"
Sep 18 13:14:52.335770 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.335732    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d6137ead809380c4ef87cd83540755ba-config\") pod \"kube-apiserver-proxy-ip-10-0-129-234.ec2.internal\" (UID: \"d6137ead809380c4ef87cd83540755ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-234.ec2.internal"
Sep 18 13:14:52.347676 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:52.347649    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-234.ec2.internal\" not found"
Sep 18 13:14:52.448134 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:52.448040    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-234.ec2.internal\" not found"
Sep 18 13:14:52.505034 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.504996    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-234.ec2.internal"
Sep 18 13:14:52.508914 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.508895    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-234.ec2.internal"
Sep 18 13:14:52.548884 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:52.548856    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-234.ec2.internal\" not found"
Sep 18 13:14:52.649688 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:52.649649    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-234.ec2.internal\" not found"
Sep 18 13:14:52.750664 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:52.750551    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-234.ec2.internal\" not found"
Sep 18 13:14:52.851770 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:52.851737    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-234.ec2.internal\" not found"
Sep 18 13:14:52.878090 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.878065    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 13:14:52.878231 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.878193    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 13:14:52.909805 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.909781    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:14:52.952307 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:52.952283    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-234.ec2.internal\" not found"
Sep 18 13:14:52.982975 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.982936    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 13:09:52 +0000 UTC" deadline="2028-07-01 17:12:57.09716831 +0000 UTC"
Sep 18 13:14:52.982975 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:52.982974    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15651h58m4.114199697s"
Sep 18 13:14:53.053258 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:53.053188    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-234.ec2.internal\" not found"
Sep 18 13:14:53.084348 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.084319    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:14:53.085302 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.085283    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:14:53.085410 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.085319    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:14:53.085410 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.085336    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:14:53.122911 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:53.122866    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90c85b650444ce894d74fab7d1e58d1.slice/crio-5c37dd7e3d44a6b5c42b393a9b61c15cba9a21d11baa68e722790f3e706988be WatchSource:0}: Error finding container 5c37dd7e3d44a6b5c42b393a9b61c15cba9a21d11baa68e722790f3e706988be: Status 404 returned error can't find the container with id 5c37dd7e3d44a6b5c42b393a9b61c15cba9a21d11baa68e722790f3e706988be
Sep 18 13:14:53.123155 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:53.123135    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6137ead809380c4ef87cd83540755ba.slice/crio-244f8ea30f5358d09acca257cfcfdd2de967edfb2721694b71744311b0359ff8 WatchSource:0}: Error finding container 244f8ea30f5358d09acca257cfcfdd2de967edfb2721694b71744311b0359ff8: Status 404 returned error can't find the container with id 244f8ea30f5358d09acca257cfcfdd2de967edfb2721694b71744311b0359ff8
Sep 18 13:14:53.129219 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.129201    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 13:14:53.151602 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.151567    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:14:53.238499 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.238452    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-234.ec2.internal"
Sep 18 13:14:53.253945 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.253924    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 13:14:53.254981 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.254967    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-234.ec2.internal"
Sep 18 13:14:53.265680 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.265655    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 13:14:53.484629 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.484534    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:14:53.914115 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.914031    2566 apiserver.go:52] "Watching apiserver"
Sep 18 13:14:53.924985 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.924959    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 13:14:53.925478 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.925450    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-ljpp4","kube-system/konnectivity-agent-b8g55","kube-system/kube-apiserver-proxy-ip-10-0-129-234.ec2.internal","openshift-cluster-node-tuning-operator/tuned-gcwfs","openshift-dns/node-resolver-rd9fl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-234.ec2.internal","openshift-multus/multus-4pn75","openshift-multus/network-metrics-daemon-h9kt5","openshift-network-diagnostics/network-check-target-vdj8k","kube-system/global-pull-secret-syncer-bgp65","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp","openshift-image-registry/node-ca-lglgb","openshift-multus/multus-additional-cni-plugins-t7jvx","openshift-network-operator/iptables-alerter-vrpqk"]
Sep 18 13:14:53.928227 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.928195    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:14:53.928351 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:53.928302    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdj8k" podUID="56a3b9a4-38a4-4412-9876-95c3c51c8aff"
Sep 18 13:14:53.930585 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.930559    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b8g55"
Sep 18 13:14:53.933455 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.932910    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 13:14:53.933455 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.932959    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp"
Sep 18 13:14:53.935613 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.935580    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:53.937712 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.937657    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 13:14:53.937933 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.937907    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4pn75"
Sep 18 13:14:53.938225 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.937916    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 13:14:53.938463 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.938440    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 13:14:53.940920 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.940443    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:14:53.940920 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.940551    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zxldr\""
Sep 18 13:14:53.940920 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.940863    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 13:14:53.940920 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.940887    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 13:14:53.941134 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.941094    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 13:14:53.941134 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.941121    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 13:14:53.941247 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.941179    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 13:14:53.941303 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.941274    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2blz6\""
Sep 18 13:14:53.941358 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.941303    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 13:14:53.941411 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.941363    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:53.941411 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.941375    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6zks5\""
Sep 18 13:14:53.941536 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.941516    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9rlwz\""
Sep 18 13:14:53.941536 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.941547    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 13:14:53.943392 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.943119    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:14:53.943392 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:53.943188    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9kt5" podUID="e7608b5b-a581-407b-94b9-3bd63a76c9f0"
Sep 18 13:14:53.943735 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.943710    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-lib-modules\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:53.943841 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.943753    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqn77\" (UniqueName: \"kubernetes.io/projected/56a3b9a4-38a4-4412-9876-95c3c51c8aff-kube-api-access-lqn77\") pod \"network-check-target-vdj8k\" (UID: \"56a3b9a4-38a4-4412-9876-95c3c51c8aff\") " pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:14:53.943841 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.943787    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/935f543f-0680-458e-a537-9172ede6c7dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z94bp\" (UID: \"935f543f-0680-458e-a537-9172ede6c7dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp"
Sep 18 13:14:53.943841 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.943812    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-etc-sysctl-conf\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:53.943841 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.943835    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-sys\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:53.943998 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.943860    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-host\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:53.943998 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.943883    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c7405cce-1c81-4043-974b-ea69248ca9cb-etc-tuned\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:53.943998 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.943913    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q54d\" (UniqueName: \"kubernetes.io/projected/c7405cce-1c81-4043-974b-ea69248ca9cb-kube-api-access-8q54d\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:53.943998 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.943942    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/df010623-6a98-476b-9bd6-26146ad788c8-agent-certs\") pod \"konnectivity-agent-b8g55\" (UID: \"df010623-6a98-476b-9bd6-26146ad788c8\") " pod="kube-system/konnectivity-agent-b8g55"
Sep 18 13:14:53.943998 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.943966    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/935f543f-0680-458e-a537-9172ede6c7dd-device-dir\") pod \"aws-ebs-csi-driver-node-z94bp\" (UID: \"935f543f-0680-458e-a537-9172ede6c7dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp"
Sep 18 13:14:53.944255 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.944007    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/935f543f-0680-458e-a537-9172ede6c7dd-sys-fs\") pod \"aws-ebs-csi-driver-node-z94bp\" (UID: \"935f543f-0680-458e-a537-9172ede6c7dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp"
Sep 18 13:14:53.944255 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.944036    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pbsv\" (UniqueName: \"kubernetes.io/projected/935f543f-0680-458e-a537-9172ede6c7dd-kube-api-access-5pbsv\") pod \"aws-ebs-csi-driver-node-z94bp\" (UID: \"935f543f-0680-458e-a537-9172ede6c7dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp"
Sep 18 13:14:53.944255 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.944062    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-etc-systemd\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:53.944255 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.944083    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-run\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:53.944255 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.944133    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7405cce-1c81-4043-974b-ea69248ca9cb-tmp\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:53.944255 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.944173    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/935f543f-0680-458e-a537-9172ede6c7dd-socket-dir\") pod \"aws-ebs-csi-driver-node-z94bp\" (UID: \"935f543f-0680-458e-a537-9172ede6c7dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp"
Sep 18 13:14:53.944255 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.944217    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/935f543f-0680-458e-a537-9172ede6c7dd-registration-dir\") pod \"aws-ebs-csi-driver-node-z94bp\" (UID: \"935f543f-0680-458e-a537-9172ede6c7dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp"
Sep 18 13:14:53.944255 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.944248    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-etc-sysconfig\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:53.944652 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.944272    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-etc-kubernetes\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:53.944652 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.944295    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-etc-sysctl-d\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:53.944652 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.944319    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-var-lib-kubelet\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:53.944652 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.944343    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/df010623-6a98-476b-9bd6-26146ad788c8-konnectivity-ca\") pod \"konnectivity-agent-b8g55\" (UID: \"df010623-6a98-476b-9bd6-26146ad788c8\") " pod="kube-system/konnectivity-agent-b8g55"
Sep 18 13:14:53.944652 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.944369    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/935f543f-0680-458e-a537-9172ede6c7dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-z94bp\" (UID: \"935f543f-0680-458e-a537-9172ede6c7dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp"
Sep 18 13:14:53.944652 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.944393    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-etc-modprobe-d\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:53.945540 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.945518    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rd9fl"
Sep 18 13:14:53.947789 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.947533    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 13:14:53.947789 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.947570    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 13:14:53.949000 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.948846    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:14:53.949000 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:53.948921    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgp65" podUID="15a0691d-d5b4-41d9-a89c-705ee52f7b25"
Sep 18 13:14:53.949176 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.949151    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 13:14:53.949622 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.949602    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 13:14:53.949849 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.949827    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 13:14:53.950838 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.950598    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 13:14:53.950838 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.950627    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 13:14:53.950838 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.950792    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qcxdz\""
Sep 18 13:14:53.950991 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.950601    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:14:53.951142 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.951121    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 13:14:53.951210 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.951162    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zb4hw\""
Sep 18 13:14:53.953012 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.952748    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lglgb"
Sep 18 13:14:53.955802 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.955782    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t7jvx"
Sep 18 13:14:53.956268 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.956243    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 13:14:53.958171 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.957828    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 13:14:53.958171 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.958073    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 13:14:53.958404 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.958386    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vrpqk"
Sep 18 13:14:53.958925 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.958904    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 13:14:53.960602 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.960582    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-krbtm\""
Sep 18 13:14:53.962230 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.962209    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 13:14:53.962344 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.962258    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 13:14:53.962405 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.962371    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:14:53.962460 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.962437    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 13:14:53.962512 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.962473    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wfpgq\""
Sep 18 13:14:53.962566 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.962508    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 13:14:53.962725 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:53.962697    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vkb2b\""
Sep 18 13:14:54.005213 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.005187    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-btzjb"
Sep 18 13:14:54.017362 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.017338    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-btzjb"
Sep 18 13:14:54.034806 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.034785    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 13:14:54.044611 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.044563    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-host-run-netns\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.044757 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.044631    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-etc-sysctl-conf\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.044757 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.044659    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c7405cce-1c81-4043-974b-ea69248ca9cb-etc-tuned\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.044757 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.044719    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-multus-cni-dir\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.044757 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.044748    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-host-kubelet\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.044982 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.044776    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f82b698-4170-4111-8be8-91ee63365aeb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t7jvx\" (UID: \"0f82b698-4170-4111-8be8-91ee63365aeb\") " pod="openshift-multus/multus-additional-cni-plugins-t7jvx"
Sep 18 13:14:54.044982 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.044794    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-etc-sysctl-conf\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.044982 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.044800    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-run-systemd\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.044982 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.044864    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/935f543f-0680-458e-a537-9172ede6c7dd-device-dir\") pod \"aws-ebs-csi-driver-node-z94bp\" (UID: \"935f543f-0680-458e-a537-9172ede6c7dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp"
Sep 18 13:14:54.044982 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.044894    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/935f543f-0680-458e-a537-9172ede6c7dd-sys-fs\") pod \"aws-ebs-csi-driver-node-z94bp\" (UID: \"935f543f-0680-458e-a537-9172ede6c7dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp"
Sep 18 13:14:54.044982 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.044940    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-etc-systemd\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.044982 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.044964    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-cnibin\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.044982 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.044984    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06c94dfa-d0af-4756-9cb6-65795af43490-cni-binary-copy\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.045410 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045007    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-systemd-units\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.045410 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045051    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c6e79ce7-2e57-4103-871f-5b11f0a468d5-hosts-file\") pod \"node-resolver-rd9fl\" (UID: \"c6e79ce7-2e57-4103-871f-5b11f0a468d5\") " pod="openshift-dns/node-resolver-rd9fl"
Sep 18 13:14:54.045410 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045075    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f82b698-4170-4111-8be8-91ee63365aeb-cni-binary-copy\") pod \"multus-additional-cni-plugins-t7jvx\" (UID: \"0f82b698-4170-4111-8be8-91ee63365aeb\") " pod="openshift-multus/multus-additional-cni-plugins-t7jvx"
Sep 18 13:14:54.045410 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045113    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqn77\" (UniqueName: \"kubernetes.io/projected/56a3b9a4-38a4-4412-9876-95c3c51c8aff-kube-api-access-lqn77\") pod \"network-check-target-vdj8k\" (UID: \"56a3b9a4-38a4-4412-9876-95c3c51c8aff\") " pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:14:54.045410 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045136    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-host-var-lib-kubelet\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.045410 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045155    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-hostroot\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.045410 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045176    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k44m8\" (UniqueName: \"kubernetes.io/projected/8a55b8ec-bd0a-4751-bc08-10e38bf2cc8f-kube-api-access-k44m8\") pod \"iptables-alerter-vrpqk\" (UID: \"8a55b8ec-bd0a-4751-bc08-10e38bf2cc8f\") " pod="openshift-network-operator/iptables-alerter-vrpqk"
Sep 18 13:14:54.045410 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045204    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-host\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.045410 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045224    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-system-cni-dir\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.045410 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045310    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-host-cni-bin\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.045410 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045333    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlfs8\" (UniqueName: \"kubernetes.io/projected/c6e79ce7-2e57-4103-871f-5b11f0a468d5-kube-api-access-jlfs8\") pod \"node-resolver-rd9fl\" (UID: \"c6e79ce7-2e57-4103-871f-5b11f0a468d5\") " pod="openshift-dns/node-resolver-rd9fl"
Sep 18 13:14:54.045410 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045354    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/df2366e7-fd3e-49d3-94af-686a9a9f262f-serviceca\") pod \"node-ca-lglgb\" (UID: \"df2366e7-fd3e-49d3-94af-686a9a9f262f\") " pod="openshift-image-registry/node-ca-lglgb"
Sep 18 13:14:54.045410 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045378    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/df010623-6a98-476b-9bd6-26146ad788c8-agent-certs\") pod \"konnectivity-agent-b8g55\" (UID: \"df010623-6a98-476b-9bd6-26146ad788c8\") " pod="kube-system/konnectivity-agent-b8g55"
Sep 18 13:14:54.046112 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045411    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-multus-socket-dir-parent\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.046112 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045452    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/935f543f-0680-458e-a537-9172ede6c7dd-registration-dir\") pod \"aws-ebs-csi-driver-node-z94bp\" (UID: \"935f543f-0680-458e-a537-9172ede6c7dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp"
Sep 18 13:14:54.046112 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045476    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-node-log\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.046112 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045481    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/935f543f-0680-458e-a537-9172ede6c7dd-sys-fs\") pod \"aws-ebs-csi-driver-node-z94bp\" (UID: \"935f543f-0680-458e-a537-9172ede6c7dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp"
Sep 18 13:14:54.046112 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045510    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-host-cni-netd\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.046112 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045517    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/935f543f-0680-458e-a537-9172ede6c7dd-device-dir\") pod \"aws-ebs-csi-driver-node-z94bp\" (UID: \"935f543f-0680-458e-a537-9172ede6c7dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp"
Sep 18 13:14:54.046112 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045513    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 13:14:54.046112 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045539    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.046112 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045626    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-host\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.046112 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045693    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/935f543f-0680-458e-a537-9172ede6c7dd-registration-dir\") pod \"aws-ebs-csi-driver-node-z94bp\" (UID: \"935f543f-0680-458e-a537-9172ede6c7dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp"
Sep 18 13:14:54.046112 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045727    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/15a0691d-d5b4-41d9-a89c-705ee52f7b25-dbus\") pod \"global-pull-secret-syncer-bgp65\" (UID: \"15a0691d-d5b4-41d9-a89c-705ee52f7b25\") " pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:14:54.046112 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045755    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-etc-modprobe-d\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.046112 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045780    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8836ae4c-f154-4c70-b955-1511e2f6a979-env-overrides\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.046112 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045814    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7608b5b-a581-407b-94b9-3bd63a76c9f0-metrics-certs\") pod \"network-metrics-daemon-h9kt5\" (UID: \"e7608b5b-a581-407b-94b9-3bd63a76c9f0\") " pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:14:54.046112 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045840    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-run-openvswitch\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.046112 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.045934    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-etc-systemd\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.046112 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046102    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-sys\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.046911 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046130    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q54d\" (UniqueName: \"kubernetes.io/projected/c7405cce-1c81-4043-974b-ea69248ca9cb-kube-api-access-8q54d\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.046911 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046161    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-host-var-lib-cni-multus\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.046911 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046207    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfphq\" (UniqueName: \"kubernetes.io/projected/e7608b5b-a581-407b-94b9-3bd63a76c9f0-kube-api-access-zfphq\") pod \"network-metrics-daemon-h9kt5\" (UID: \"e7608b5b-a581-407b-94b9-3bd63a76c9f0\") " pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:14:54.046911 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046243    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15a0691d-d5b4-41d9-a89c-705ee52f7b25-original-pull-secret\") pod \"global-pull-secret-syncer-bgp65\" (UID: \"15a0691d-d5b4-41d9-a89c-705ee52f7b25\") " pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:14:54.046911 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046295    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f82b698-4170-4111-8be8-91ee63365aeb-system-cni-dir\") pod \"multus-additional-cni-plugins-t7jvx\" (UID: \"0f82b698-4170-4111-8be8-91ee63365aeb\") " pod="openshift-multus/multus-additional-cni-plugins-t7jvx"
Sep 18 13:14:54.046911 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046320    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-sys\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.046911 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046356    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-host-run-ovn-kubernetes\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.046911 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046385    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pbsv\" (UniqueName: \"kubernetes.io/projected/935f543f-0680-458e-a537-9172ede6c7dd-kube-api-access-5pbsv\") pod \"aws-ebs-csi-driver-node-z94bp\" (UID: \"935f543f-0680-458e-a537-9172ede6c7dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp"
Sep 18 13:14:54.046911 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046429    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-run\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.046911 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046480    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-etc-modprobe-d\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.046911 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046490    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-run\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.046911 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046504    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pm69\" (UniqueName: \"kubernetes.io/projected/0f82b698-4170-4111-8be8-91ee63365aeb-kube-api-access-8pm69\") pod \"multus-additional-cni-plugins-t7jvx\" (UID: \"0f82b698-4170-4111-8be8-91ee63365aeb\") " pod="openshift-multus/multus-additional-cni-plugins-t7jvx"
Sep 18 13:14:54.046911 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046537    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a55b8ec-bd0a-4751-bc08-10e38bf2cc8f-iptables-alerter-script\") pod \"iptables-alerter-vrpqk\" (UID: \"8a55b8ec-bd0a-4751-bc08-10e38bf2cc8f\") " pod="openshift-network-operator/iptables-alerter-vrpqk"
Sep 18 13:14:54.046911 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046574    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/935f543f-0680-458e-a537-9172ede6c7dd-socket-dir\") pod \"aws-ebs-csi-driver-node-z94bp\" (UID: \"935f543f-0680-458e-a537-9172ede6c7dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp"
Sep 18 13:14:54.046911 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046601    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-etc-kubernetes\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.046911 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046649    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-host-slash\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.047572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046673    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-etc-openvswitch\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.047572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046696    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8836ae4c-f154-4c70-b955-1511e2f6a979-ovn-node-metrics-cert\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.047572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046722    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8836ae4c-f154-4c70-b955-1511e2f6a979-ovnkube-script-lib\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.047572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046733    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/935f543f-0680-458e-a537-9172ede6c7dd-socket-dir\") pod \"aws-ebs-csi-driver-node-z94bp\" (UID: \"935f543f-0680-458e-a537-9172ede6c7dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp"
Sep 18 13:14:54.047572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046747    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df2366e7-fd3e-49d3-94af-686a9a9f262f-host\") pod \"node-ca-lglgb\" (UID: \"df2366e7-fd3e-49d3-94af-686a9a9f262f\") " pod="openshift-image-registry/node-ca-lglgb"
Sep 18 13:14:54.047572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/df010623-6a98-476b-9bd6-26146ad788c8-konnectivity-ca\") pod \"konnectivity-agent-b8g55\" (UID: \"df010623-6a98-476b-9bd6-26146ad788c8\") " pod="kube-system/konnectivity-agent-b8g55"
Sep 18 13:14:54.047572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046799    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-lib-modules\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.047572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046822    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-multus-conf-dir\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.047572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046843    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-etc-kubernetes\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.047572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046847    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-host-run-multus-certs\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.047572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046883    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-etc-kubernetes\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.047572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f82b698-4170-4111-8be8-91ee63365aeb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t7jvx\" (UID: \"0f82b698-4170-4111-8be8-91ee63365aeb\") " pod="openshift-multus/multus-additional-cni-plugins-t7jvx"
Sep 18 13:14:54.047572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/935f543f-0680-458e-a537-9172ede6c7dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z94bp\" (UID: \"935f543f-0680-458e-a537-9172ede6c7dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp"
Sep 18 13:14:54.047572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.046971    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-var-lib-openvswitch\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.047572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047012    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/15a0691d-d5b4-41d9-a89c-705ee52f7b25-kubelet-config\") pod \"global-pull-secret-syncer-bgp65\" (UID: \"15a0691d-d5b4-41d9-a89c-705ee52f7b25\") " pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:14:54.047572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047037    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72qgd\" (UniqueName: \"kubernetes.io/projected/df2366e7-fd3e-49d3-94af-686a9a9f262f-kube-api-access-72qgd\") pod \"node-ca-lglgb\" (UID: \"df2366e7-fd3e-49d3-94af-686a9a9f262f\") " pod="openshift-image-registry/node-ca-lglgb"
Sep 18 13:14:54.047572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047061    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7405cce-1c81-4043-974b-ea69248ca9cb-tmp\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.048464 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047082    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-host-run-k8s-cni-cncf-io\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.048464 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047102    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-log-socket\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.048464 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047121    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e79ce7-2e57-4103-871f-5b11f0a468d5-tmp-dir\") pod \"node-resolver-rd9fl\" (UID: \"c6e79ce7-2e57-4103-871f-5b11f0a468d5\") " pod="openshift-dns/node-resolver-rd9fl"
Sep 18 13:14:54.048464 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047141    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a55b8ec-bd0a-4751-bc08-10e38bf2cc8f-host-slash\") pod \"iptables-alerter-vrpqk\" (UID: \"8a55b8ec-bd0a-4751-bc08-10e38bf2cc8f\") " pod="openshift-network-operator/iptables-alerter-vrpqk"
Sep 18 13:14:54.048464 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047162    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-etc-sysconfig\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.048464 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047182    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-etc-sysctl-d\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.048464 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047203    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-var-lib-kubelet\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.048464 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047208    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-lib-modules\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.048464 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047231    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jwhb\" (UniqueName: \"kubernetes.io/projected/06c94dfa-d0af-4756-9cb6-65795af43490-kube-api-access-8jwhb\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.048464 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047252    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hftdj\" (UniqueName: \"kubernetes.io/projected/8836ae4c-f154-4c70-b955-1511e2f6a979-kube-api-access-hftdj\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.048464 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047274    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f82b698-4170-4111-8be8-91ee63365aeb-cnibin\") pod \"multus-additional-cni-plugins-t7jvx\" (UID: \"0f82b698-4170-4111-8be8-91ee63365aeb\") " pod="openshift-multus/multus-additional-cni-plugins-t7jvx"
Sep 18 13:14:54.048464 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047299    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f82b698-4170-4111-8be8-91ee63365aeb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t7jvx\" (UID: \"0f82b698-4170-4111-8be8-91ee63365aeb\") " pod="openshift-multus/multus-additional-cni-plugins-t7jvx"
Sep 18 13:14:54.048464 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047309    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/935f543f-0680-458e-a537-9172ede6c7dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z94bp\" (UID: \"935f543f-0680-458e-a537-9172ede6c7dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp"
Sep 18 13:14:54.048464 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047325    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/935f543f-0680-458e-a537-9172ede6c7dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-z94bp\" (UID: \"935f543f-0680-458e-a537-9172ede6c7dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp"
Sep 18 13:14:54.048464 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047378    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-host-run-netns\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.048464 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047404    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-host-var-lib-cni-bin\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.048464 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047405    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-var-lib-kubelet\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.049417 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047411    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-etc-sysconfig\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.049417 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047446    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/06c94dfa-d0af-4756-9cb6-65795af43490-multus-daemon-config\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.049417 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047470    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8836ae4c-f154-4c70-b955-1511e2f6a979-ovnkube-config\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.049417 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047495    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-run-ovn\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.049417 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047513    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/935f543f-0680-458e-a537-9172ede6c7dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-z94bp\" (UID: \"935f543f-0680-458e-a537-9172ede6c7dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp"
Sep 18 13:14:54.049417 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047500    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c7405cce-1c81-4043-974b-ea69248ca9cb-etc-sysctl-d\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.049417 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.047662    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/df010623-6a98-476b-9bd6-26146ad788c8-konnectivity-ca\") pod \"konnectivity-agent-b8g55\" (UID: \"df010623-6a98-476b-9bd6-26146ad788c8\") " pod="kube-system/konnectivity-agent-b8g55"
Sep 18 13:14:54.049770 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.049422    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c7405cce-1c81-4043-974b-ea69248ca9cb-etc-tuned\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.049770 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.049758    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/df010623-6a98-476b-9bd6-26146ad788c8-agent-certs\") pod \"konnectivity-agent-b8g55\" (UID: \"df010623-6a98-476b-9bd6-26146ad788c8\") " pod="kube-system/konnectivity-agent-b8g55"
Sep 18 13:14:54.049880 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.049852    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7405cce-1c81-4043-974b-ea69248ca9cb-tmp\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.055800 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:54.055776    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:14:54.055907 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:54.055803    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:14:54.055907 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:54.055821    2566 projected.go:196] Error preparing data for projected volume kube-api-access-lqn77 for pod openshift-network-diagnostics/network-check-target-vdj8k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:14:54.055907 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:54.055887    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56a3b9a4-38a4-4412-9876-95c3c51c8aff-kube-api-access-lqn77 podName:56a3b9a4-38a4-4412-9876-95c3c51c8aff nodeName:}" failed. No retries permitted until 2026-09-18 13:14:54.555871221 +0000 UTC m=+2.990747605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lqn77" (UniqueName: "kubernetes.io/projected/56a3b9a4-38a4-4412-9876-95c3c51c8aff-kube-api-access-lqn77") pod "network-check-target-vdj8k" (UID: "56a3b9a4-38a4-4412-9876-95c3c51c8aff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:14:54.059149 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.059128    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q54d\" (UniqueName: \"kubernetes.io/projected/c7405cce-1c81-4043-974b-ea69248ca9cb-kube-api-access-8q54d\") pod \"tuned-gcwfs\" (UID: \"c7405cce-1c81-4043-974b-ea69248ca9cb\") " pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.062866 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.062842    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pbsv\" (UniqueName: \"kubernetes.io/projected/935f543f-0680-458e-a537-9172ede6c7dd-kube-api-access-5pbsv\") pod \"aws-ebs-csi-driver-node-z94bp\" (UID: \"935f543f-0680-458e-a537-9172ede6c7dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp"
Sep 18 13:14:54.075826 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.075764    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-234.ec2.internal" event={"ID":"d6137ead809380c4ef87cd83540755ba","Type":"ContainerStarted","Data":"244f8ea30f5358d09acca257cfcfdd2de967edfb2721694b71744311b0359ff8"}
Sep 18 13:14:54.077243 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.077199    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-234.ec2.internal" event={"ID":"e90c85b650444ce894d74fab7d1e58d1","Type":"ContainerStarted","Data":"5c37dd7e3d44a6b5c42b393a9b61c15cba9a21d11baa68e722790f3e706988be"}
Sep 18 13:14:54.148007 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.147964    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f82b698-4170-4111-8be8-91ee63365aeb-system-cni-dir\") pod \"multus-additional-cni-plugins-t7jvx\" (UID: \"0f82b698-4170-4111-8be8-91ee63365aeb\") " pod="openshift-multus/multus-additional-cni-plugins-t7jvx"
Sep 18 13:14:54.148007 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148001    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-host-run-ovn-kubernetes\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.148253 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148021    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pm69\" (UniqueName: \"kubernetes.io/projected/0f82b698-4170-4111-8be8-91ee63365aeb-kube-api-access-8pm69\") pod \"multus-additional-cni-plugins-t7jvx\" (UID: \"0f82b698-4170-4111-8be8-91ee63365aeb\") " pod="openshift-multus/multus-additional-cni-plugins-t7jvx"
Sep 18 13:14:54.148253 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148047    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a55b8ec-bd0a-4751-bc08-10e38bf2cc8f-iptables-alerter-script\") pod \"iptables-alerter-vrpqk\" (UID: \"8a55b8ec-bd0a-4751-bc08-10e38bf2cc8f\") " pod="openshift-network-operator/iptables-alerter-vrpqk"
Sep 18 13:14:54.148253 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148072    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-host-slash\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.148253 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148091    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f82b698-4170-4111-8be8-91ee63365aeb-system-cni-dir\") pod \"multus-additional-cni-plugins-t7jvx\" (UID: \"0f82b698-4170-4111-8be8-91ee63365aeb\") " pod="openshift-multus/multus-additional-cni-plugins-t7jvx"
Sep 18 13:14:54.148253 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148106    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-host-run-ovn-kubernetes\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.148253 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148097    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-etc-openvswitch\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.148253 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148129    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-etc-openvswitch\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.148253 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148152    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8836ae4c-f154-4c70-b955-1511e2f6a979-ovn-node-metrics-cert\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.148253 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148215    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-host-slash\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.148758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148339    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8836ae4c-f154-4c70-b955-1511e2f6a979-ovnkube-script-lib\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.148758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148371    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df2366e7-fd3e-49d3-94af-686a9a9f262f-host\") pod \"node-ca-lglgb\" (UID: \"df2366e7-fd3e-49d3-94af-686a9a9f262f\") " pod="openshift-image-registry/node-ca-lglgb"
Sep 18 13:14:54.148758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148397    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-multus-conf-dir\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.148758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-host-run-multus-certs\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.148758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148451    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-etc-kubernetes\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.148758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148455    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df2366e7-fd3e-49d3-94af-686a9a9f262f-host\") pod \"node-ca-lglgb\" (UID: \"df2366e7-fd3e-49d3-94af-686a9a9f262f\") " pod="openshift-image-registry/node-ca-lglgb"
Sep 18 13:14:54.148758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148477    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f82b698-4170-4111-8be8-91ee63365aeb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t7jvx\" (UID: \"0f82b698-4170-4111-8be8-91ee63365aeb\") " pod="openshift-multus/multus-additional-cni-plugins-t7jvx"
Sep 18 13:14:54.148758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148521    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-var-lib-openvswitch\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.148758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148550    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/15a0691d-d5b4-41d9-a89c-705ee52f7b25-kubelet-config\") pod \"global-pull-secret-syncer-bgp65\" (UID: \"15a0691d-d5b4-41d9-a89c-705ee52f7b25\") " pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:14:54.148758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148577    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72qgd\" (UniqueName: \"kubernetes.io/projected/df2366e7-fd3e-49d3-94af-686a9a9f262f-kube-api-access-72qgd\") pod \"node-ca-lglgb\" (UID: \"df2366e7-fd3e-49d3-94af-686a9a9f262f\") " pod="openshift-image-registry/node-ca-lglgb"
Sep 18 13:14:54.148758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148633    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-host-run-multus-certs\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.148758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148631    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-var-lib-openvswitch\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.148758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148685    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-multus-conf-dir\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.148758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148687    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-host-run-k8s-cni-cncf-io\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.148758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148719    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-log-socket\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.148758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148720    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-host-run-k8s-cni-cncf-io\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.148758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148747    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e79ce7-2e57-4103-871f-5b11f0a468d5-tmp-dir\") pod \"node-resolver-rd9fl\" (UID: \"c6e79ce7-2e57-4103-871f-5b11f0a468d5\") " pod="openshift-dns/node-resolver-rd9fl"
Sep 18 13:14:54.149624 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148773    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-log-socket\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.149624 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148811    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a55b8ec-bd0a-4751-bc08-10e38bf2cc8f-host-slash\") pod \"iptables-alerter-vrpqk\" (UID: \"8a55b8ec-bd0a-4751-bc08-10e38bf2cc8f\") " pod="openshift-network-operator/iptables-alerter-vrpqk"
Sep 18 13:14:54.149624 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148853    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jwhb\" (UniqueName: \"kubernetes.io/projected/06c94dfa-d0af-4756-9cb6-65795af43490-kube-api-access-8jwhb\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.149624 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148867    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a55b8ec-bd0a-4751-bc08-10e38bf2cc8f-host-slash\") pod \"iptables-alerter-vrpqk\" (UID: \"8a55b8ec-bd0a-4751-bc08-10e38bf2cc8f\") " pod="openshift-network-operator/iptables-alerter-vrpqk"
Sep 18 13:14:54.149624 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148869    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/15a0691d-d5b4-41d9-a89c-705ee52f7b25-kubelet-config\") pod \"global-pull-secret-syncer-bgp65\" (UID: \"15a0691d-d5b4-41d9-a89c-705ee52f7b25\") " pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:14:54.149624 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148885    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-etc-kubernetes\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.149624 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148816    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a55b8ec-bd0a-4751-bc08-10e38bf2cc8f-iptables-alerter-script\") pod \"iptables-alerter-vrpqk\" (UID: \"8a55b8ec-bd0a-4751-bc08-10e38bf2cc8f\") " pod="openshift-network-operator/iptables-alerter-vrpqk"
Sep 18 13:14:54.149624 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148880    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hftdj\" (UniqueName: \"kubernetes.io/projected/8836ae4c-f154-4c70-b955-1511e2f6a979-kube-api-access-hftdj\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.149624 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148948    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f82b698-4170-4111-8be8-91ee63365aeb-cnibin\") pod \"multus-additional-cni-plugins-t7jvx\" (UID: \"0f82b698-4170-4111-8be8-91ee63365aeb\") " pod="openshift-multus/multus-additional-cni-plugins-t7jvx"
Sep 18 13:14:54.149624 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.148975    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f82b698-4170-4111-8be8-91ee63365aeb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t7jvx\" (UID: \"0f82b698-4170-4111-8be8-91ee63365aeb\") " pod="openshift-multus/multus-additional-cni-plugins-t7jvx"
Sep 18 13:14:54.149624 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149003    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-host-run-netns\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.149624 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149026    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-host-var-lib-cni-bin\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.149624 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149045    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f82b698-4170-4111-8be8-91ee63365aeb-cnibin\") pod \"multus-additional-cni-plugins-t7jvx\" (UID: \"0f82b698-4170-4111-8be8-91ee63365aeb\") " pod="openshift-multus/multus-additional-cni-plugins-t7jvx"
Sep 18 13:14:54.149624 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149050    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/06c94dfa-d0af-4756-9cb6-65795af43490-multus-daemon-config\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.149624 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149074    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f82b698-4170-4111-8be8-91ee63365aeb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t7jvx\" (UID: \"0f82b698-4170-4111-8be8-91ee63365aeb\") " pod="openshift-multus/multus-additional-cni-plugins-t7jvx"
Sep 18 13:14:54.149624 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149068    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8836ae4c-f154-4c70-b955-1511e2f6a979-ovnkube-script-lib\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.149624 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149094    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8836ae4c-f154-4c70-b955-1511e2f6a979-ovnkube-config\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.150517 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149127    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6e79ce7-2e57-4103-871f-5b11f0a468d5-tmp-dir\") pod \"node-resolver-rd9fl\" (UID: \"c6e79ce7-2e57-4103-871f-5b11f0a468d5\") " pod="openshift-dns/node-resolver-rd9fl"
Sep 18 13:14:54.150517 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149147    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-run-ovn\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.150517 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149160    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-host-var-lib-cni-bin\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.150517 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149176    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-host-run-netns\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.150517 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149202    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-run-ovn\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.150517 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149207    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-multus-cni-dir\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.150517 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149231    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-host-kubelet\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.150517 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149245    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-host-run-netns\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.150517 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149257    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f82b698-4170-4111-8be8-91ee63365aeb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t7jvx\" (UID: \"0f82b698-4170-4111-8be8-91ee63365aeb\") " pod="openshift-multus/multus-additional-cni-plugins-t7jvx"
Sep 18 13:14:54.150517 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149282    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-run-systemd\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.150517 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-cnibin\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.150517 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149332    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06c94dfa-d0af-4756-9cb6-65795af43490-cni-binary-copy\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.150517 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149366    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-systemd-units\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.150517 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149393    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c6e79ce7-2e57-4103-871f-5b11f0a468d5-hosts-file\") pod \"node-resolver-rd9fl\" (UID: \"c6e79ce7-2e57-4103-871f-5b11f0a468d5\") " pod="openshift-dns/node-resolver-rd9fl"
Sep 18 13:14:54.150517 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149401    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-multus-cni-dir\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.150517 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149421    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f82b698-4170-4111-8be8-91ee63365aeb-cni-binary-copy\") pod \"multus-additional-cni-plugins-t7jvx\" (UID: \"0f82b698-4170-4111-8be8-91ee63365aeb\") " pod="openshift-multus/multus-additional-cni-plugins-t7jvx"
Sep 18 13:14:54.150517 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149459    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-host-var-lib-kubelet\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.150517 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149482    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-hostroot\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.151264 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149507    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k44m8\" (UniqueName: \"kubernetes.io/projected/8a55b8ec-bd0a-4751-bc08-10e38bf2cc8f-kube-api-access-k44m8\") pod \"iptables-alerter-vrpqk\" (UID: \"8a55b8ec-bd0a-4751-bc08-10e38bf2cc8f\") " pod="openshift-network-operator/iptables-alerter-vrpqk"
Sep 18 13:14:54.151264 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149511    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f82b698-4170-4111-8be8-91ee63365aeb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t7jvx\" (UID: \"0f82b698-4170-4111-8be8-91ee63365aeb\") " pod="openshift-multus/multus-additional-cni-plugins-t7jvx"
Sep 18 13:14:54.151264 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149533    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-system-cni-dir\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.151264 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149558    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-host-cni-bin\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.151264 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149577    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8836ae4c-f154-4c70-b955-1511e2f6a979-ovnkube-config\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.151264 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149587    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlfs8\" (UniqueName: \"kubernetes.io/projected/c6e79ce7-2e57-4103-871f-5b11f0a468d5-kube-api-access-jlfs8\") pod \"node-resolver-rd9fl\" (UID: \"c6e79ce7-2e57-4103-871f-5b11f0a468d5\") " pod="openshift-dns/node-resolver-rd9fl"
Sep 18 13:14:54.151264 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149593    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-run-systemd\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.151264 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149613    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/06c94dfa-d0af-4756-9cb6-65795af43490-multus-daemon-config\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.151264 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149624    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/df2366e7-fd3e-49d3-94af-686a9a9f262f-serviceca\") pod \"node-ca-lglgb\" (UID: \"df2366e7-fd3e-49d3-94af-686a9a9f262f\") " pod="openshift-image-registry/node-ca-lglgb"
Sep 18 13:14:54.151264 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149635    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-host-run-netns\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.151264 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149665    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-multus-socket-dir-parent\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.151264 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149709    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-cnibin\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.151264 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149714    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-systemd-units\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.151264 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149613    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f82b698-4170-4111-8be8-91ee63365aeb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t7jvx\" (UID: \"0f82b698-4170-4111-8be8-91ee63365aeb\") " pod="openshift-multus/multus-additional-cni-plugins-t7jvx"
Sep 18 13:14:54.151264 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149759    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-node-log\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.151264 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149785    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-multus-socket-dir-parent\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.151264 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149681    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-hostroot\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.151264 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149817    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c6e79ce7-2e57-4103-871f-5b11f0a468d5-hosts-file\") pod \"node-resolver-rd9fl\" (UID: \"c6e79ce7-2e57-4103-871f-5b11f0a468d5\") " pod="openshift-dns/node-resolver-rd9fl"
Sep 18 13:14:54.151953 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149716    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-node-log\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.151953 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149844    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-host-var-lib-kubelet\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.151953 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149858    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-host-cni-bin\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.151953 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149881    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-host-cni-netd\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.151953 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149561    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-host-kubelet\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.151953 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149887    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06c94dfa-d0af-4756-9cb6-65795af43490-cni-binary-copy\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.151953 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149929    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-host-cni-netd\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.151953 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149930    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.151953 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149946    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-system-cni-dir\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.151953 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149957    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/15a0691d-d5b4-41d9-a89c-705ee52f7b25-dbus\") pod \"global-pull-secret-syncer-bgp65\" (UID: \"15a0691d-d5b4-41d9-a89c-705ee52f7b25\") " pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:14:54.151953 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149982    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.151953 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.149991    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8836ae4c-f154-4c70-b955-1511e2f6a979-env-overrides\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.151953 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.150003    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/15a0691d-d5b4-41d9-a89c-705ee52f7b25-dbus\") pod \"global-pull-secret-syncer-bgp65\" (UID: \"15a0691d-d5b4-41d9-a89c-705ee52f7b25\") " pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:14:54.151953 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.150019    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7608b5b-a581-407b-94b9-3bd63a76c9f0-metrics-certs\") pod \"network-metrics-daemon-h9kt5\" (UID: \"e7608b5b-a581-407b-94b9-3bd63a76c9f0\") " pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:14:54.151953 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.150045    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-run-openvswitch\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.151953 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.150047    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/df2366e7-fd3e-49d3-94af-686a9a9f262f-serviceca\") pod \"node-ca-lglgb\" (UID: \"df2366e7-fd3e-49d3-94af-686a9a9f262f\") " pod="openshift-image-registry/node-ca-lglgb"
Sep 18 13:14:54.151953 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.150090    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8836ae4c-f154-4c70-b955-1511e2f6a979-run-openvswitch\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.151953 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:54.150119    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:14:54.152753 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.150170    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-host-var-lib-cni-multus\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.152753 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:54.150183    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7608b5b-a581-407b-94b9-3bd63a76c9f0-metrics-certs podName:e7608b5b-a581-407b-94b9-3bd63a76c9f0 nodeName:}" failed. No retries permitted until 2026-09-18 13:14:54.650166027 +0000 UTC m=+3.085042437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7608b5b-a581-407b-94b9-3bd63a76c9f0-metrics-certs") pod "network-metrics-daemon-h9kt5" (UID: "e7608b5b-a581-407b-94b9-3bd63a76c9f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:14:54.152753 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.150209    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/06c94dfa-d0af-4756-9cb6-65795af43490-host-var-lib-cni-multus\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.152753 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.150213    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfphq\" (UniqueName: \"kubernetes.io/projected/e7608b5b-a581-407b-94b9-3bd63a76c9f0-kube-api-access-zfphq\") pod \"network-metrics-daemon-h9kt5\" (UID: \"e7608b5b-a581-407b-94b9-3bd63a76c9f0\") " pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:14:54.152753 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.150245    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15a0691d-d5b4-41d9-a89c-705ee52f7b25-original-pull-secret\") pod \"global-pull-secret-syncer-bgp65\" (UID: \"15a0691d-d5b4-41d9-a89c-705ee52f7b25\") " pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:14:54.152753 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.150323    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f82b698-4170-4111-8be8-91ee63365aeb-cni-binary-copy\") pod \"multus-additional-cni-plugins-t7jvx\" (UID: \"0f82b698-4170-4111-8be8-91ee63365aeb\") " pod="openshift-multus/multus-additional-cni-plugins-t7jvx"
Sep 18 13:14:54.152753 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:54.150368    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:14:54.152753 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.150390    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8836ae4c-f154-4c70-b955-1511e2f6a979-env-overrides\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.152753 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:54.150419    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a0691d-d5b4-41d9-a89c-705ee52f7b25-original-pull-secret podName:15a0691d-d5b4-41d9-a89c-705ee52f7b25 nodeName:}" failed. No retries permitted until 2026-09-18 13:14:54.650408264 +0000 UTC m=+3.085284658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/15a0691d-d5b4-41d9-a89c-705ee52f7b25-original-pull-secret") pod "global-pull-secret-syncer-bgp65" (UID: "15a0691d-d5b4-41d9-a89c-705ee52f7b25") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:14:54.152753 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.151197    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8836ae4c-f154-4c70-b955-1511e2f6a979-ovn-node-metrics-cert\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.162395 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.162365    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jwhb\" (UniqueName: \"kubernetes.io/projected/06c94dfa-d0af-4756-9cb6-65795af43490-kube-api-access-8jwhb\") pod \"multus-4pn75\" (UID: \"06c94dfa-d0af-4756-9cb6-65795af43490\") " pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.163904 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.163879    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pm69\" (UniqueName: \"kubernetes.io/projected/0f82b698-4170-4111-8be8-91ee63365aeb-kube-api-access-8pm69\") pod \"multus-additional-cni-plugins-t7jvx\" (UID: \"0f82b698-4170-4111-8be8-91ee63365aeb\") " pod="openshift-multus/multus-additional-cni-plugins-t7jvx"
Sep 18 13:14:54.166223 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.166148    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72qgd\" (UniqueName: \"kubernetes.io/projected/df2366e7-fd3e-49d3-94af-686a9a9f262f-kube-api-access-72qgd\") pod \"node-ca-lglgb\" (UID: \"df2366e7-fd3e-49d3-94af-686a9a9f262f\") " pod="openshift-image-registry/node-ca-lglgb"
Sep 18 13:14:54.166844 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.166781    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfphq\" (UniqueName: \"kubernetes.io/projected/e7608b5b-a581-407b-94b9-3bd63a76c9f0-kube-api-access-zfphq\") pod \"network-metrics-daemon-h9kt5\" (UID: \"e7608b5b-a581-407b-94b9-3bd63a76c9f0\") " pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:14:54.166975 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.166884    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlfs8\" (UniqueName: \"kubernetes.io/projected/c6e79ce7-2e57-4103-871f-5b11f0a468d5-kube-api-access-jlfs8\") pod \"node-resolver-rd9fl\" (UID: \"c6e79ce7-2e57-4103-871f-5b11f0a468d5\") " pod="openshift-dns/node-resolver-rd9fl"
Sep 18 13:14:54.168170 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.168144    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k44m8\" (UniqueName: \"kubernetes.io/projected/8a55b8ec-bd0a-4751-bc08-10e38bf2cc8f-kube-api-access-k44m8\") pod \"iptables-alerter-vrpqk\" (UID: \"8a55b8ec-bd0a-4751-bc08-10e38bf2cc8f\") " pod="openshift-network-operator/iptables-alerter-vrpqk"
Sep 18 13:14:54.168430 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.168406    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hftdj\" (UniqueName: \"kubernetes.io/projected/8836ae4c-f154-4c70-b955-1511e2f6a979-kube-api-access-hftdj\") pod \"ovnkube-node-ljpp4\" (UID: \"8836ae4c-f154-4c70-b955-1511e2f6a979\") " pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.245737 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.245702    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b8g55"
Sep 18 13:14:54.258421 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.258396    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp"
Sep 18 13:14:54.267585 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.267558    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gcwfs"
Sep 18 13:14:54.274686 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.274665    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4pn75"
Sep 18 13:14:54.282061 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.282040    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:14:54.289985 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.289965    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rd9fl"
Sep 18 13:14:54.298939 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.298909    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lglgb"
Sep 18 13:14:54.306564 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.306537    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t7jvx"
Sep 18 13:14:54.315574 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.315556    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vrpqk"
Sep 18 13:14:54.655832 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.655735    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqn77\" (UniqueName: \"kubernetes.io/projected/56a3b9a4-38a4-4412-9876-95c3c51c8aff-kube-api-access-lqn77\") pod \"network-check-target-vdj8k\" (UID: \"56a3b9a4-38a4-4412-9876-95c3c51c8aff\") " pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:14:54.655832 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.655797    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7608b5b-a581-407b-94b9-3bd63a76c9f0-metrics-certs\") pod \"network-metrics-daemon-h9kt5\" (UID: \"e7608b5b-a581-407b-94b9-3bd63a76c9f0\") " pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:14:54.655832 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:54.655828    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15a0691d-d5b4-41d9-a89c-705ee52f7b25-original-pull-secret\") pod \"global-pull-secret-syncer-bgp65\" (UID: \"15a0691d-d5b4-41d9-a89c-705ee52f7b25\") " pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:14:54.656100 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:54.655965    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:14:54.656100 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:54.655991    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:14:54.656100 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:54.656043    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:14:54.656100 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:54.656066    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7608b5b-a581-407b-94b9-3bd63a76c9f0-metrics-certs podName:e7608b5b-a581-407b-94b9-3bd63a76c9f0 nodeName:}" failed. No retries permitted until 2026-09-18 13:14:55.65604599 +0000 UTC m=+4.090922396 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7608b5b-a581-407b-94b9-3bd63a76c9f0-metrics-certs") pod "network-metrics-daemon-h9kt5" (UID: "e7608b5b-a581-407b-94b9-3bd63a76c9f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:14:54.656314 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:54.656111    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a0691d-d5b4-41d9-a89c-705ee52f7b25-original-pull-secret podName:15a0691d-d5b4-41d9-a89c-705ee52f7b25 nodeName:}" failed. No retries permitted until 2026-09-18 13:14:55.656091553 +0000 UTC m=+4.090967956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/15a0691d-d5b4-41d9-a89c-705ee52f7b25-original-pull-secret") pod "global-pull-secret-syncer-bgp65" (UID: "15a0691d-d5b4-41d9-a89c-705ee52f7b25") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:14:54.656314 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:54.655997    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:14:54.656314 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:54.656138    2566 projected.go:196] Error preparing data for projected volume kube-api-access-lqn77 for pod openshift-network-diagnostics/network-check-target-vdj8k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:14:54.656314 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:54.656177    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56a3b9a4-38a4-4412-9876-95c3c51c8aff-kube-api-access-lqn77 podName:56a3b9a4-38a4-4412-9876-95c3c51c8aff nodeName:}" failed. No retries permitted until 2026-09-18 13:14:55.656166533 +0000 UTC m=+4.091042921 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lqn77" (UniqueName: "kubernetes.io/projected/56a3b9a4-38a4-4412-9876-95c3c51c8aff-kube-api-access-lqn77") pod "network-check-target-vdj8k" (UID: "56a3b9a4-38a4-4412-9876-95c3c51c8aff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:14:55.020680 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:55.020513    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 13:09:54 +0000 UTC" deadline="2028-04-01 13:28:43.880818465 +0000 UTC"
Sep 18 13:14:55.020680 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:55.020629    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13464h13m48.860201449s"
Sep 18 13:14:55.071330 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:55.071296    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:14:55.071330 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:55.071319    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:14:55.071522 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:55.071421    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgp65" podUID="15a0691d-d5b4-41d9-a89c-705ee52f7b25"
Sep 18 13:14:55.071568 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:55.071543    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9kt5" podUID="e7608b5b-a581-407b-94b9-3bd63a76c9f0"
Sep 18 13:14:55.135870 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:55.135828    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf010623_6a98_476b_9bd6_26146ad788c8.slice/crio-7e4d33eed6d9c4d7e2e40fdd23ce831729bf593ccd0440549bd93cf8b8f94fd3 WatchSource:0}: Error finding container 7e4d33eed6d9c4d7e2e40fdd23ce831729bf593ccd0440549bd93cf8b8f94fd3: Status 404 returned error can't find the container with id 7e4d33eed6d9c4d7e2e40fdd23ce831729bf593ccd0440549bd93cf8b8f94fd3
Sep 18 13:14:55.137039 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:55.136949    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e79ce7_2e57_4103_871f_5b11f0a468d5.slice/crio-9e3a9ee668cb6edf20e45d024adf4a1a1af0148b009613a0b42985332d3f7176 WatchSource:0}: Error finding container 9e3a9ee668cb6edf20e45d024adf4a1a1af0148b009613a0b42985332d3f7176: Status 404 returned error can't find the container with id 9e3a9ee668cb6edf20e45d024adf4a1a1af0148b009613a0b42985332d3f7176
Sep 18 13:14:55.139423 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:55.138688    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a55b8ec_bd0a_4751_bc08_10e38bf2cc8f.slice/crio-05aaa4e54d9b03590258a89920f413893c7a85e4ea37040d40a0675147c90dfe WatchSource:0}: Error finding container 05aaa4e54d9b03590258a89920f413893c7a85e4ea37040d40a0675147c90dfe: Status 404 returned error can't find the container with id 05aaa4e54d9b03590258a89920f413893c7a85e4ea37040d40a0675147c90dfe
Sep 18 13:14:55.141859 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:55.141821    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2366e7_fd3e_49d3_94af_686a9a9f262f.slice/crio-3623cfccbd34a0002c4a380021de853f8f885da0e8e0d62c06417935d8540aff WatchSource:0}: Error finding container 3623cfccbd34a0002c4a380021de853f8f885da0e8e0d62c06417935d8540aff: Status 404 returned error can't find the container with id 3623cfccbd34a0002c4a380021de853f8f885da0e8e0d62c06417935d8540aff
Sep 18 13:14:55.144517 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:55.144484    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c94dfa_d0af_4756_9cb6_65795af43490.slice/crio-675acaa277e252a6aa5674756ff1ef4a8a8acc099389ce9902a5d794eb4be5d4 WatchSource:0}: Error finding container 675acaa277e252a6aa5674756ff1ef4a8a8acc099389ce9902a5d794eb4be5d4: Status 404 returned error can't find the container with id 675acaa277e252a6aa5674756ff1ef4a8a8acc099389ce9902a5d794eb4be5d4
Sep 18 13:14:55.146657 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:55.146114    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f82b698_4170_4111_8be8_91ee63365aeb.slice/crio-6d807b2e2ad81dea4716fb3003f0a3f0d3d100fae2a7cd664b37a44b1a83b222 WatchSource:0}: Error finding container 6d807b2e2ad81dea4716fb3003f0a3f0d3d100fae2a7cd664b37a44b1a83b222: Status 404 returned error can't find the container with id 6d807b2e2ad81dea4716fb3003f0a3f0d3d100fae2a7cd664b37a44b1a83b222
Sep 18 13:14:55.148225 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:14:55.147868    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8836ae4c_f154_4c70_b955_1511e2f6a979.slice/crio-9031a5946b97c4b0ae20acfff98cf2b58ce51fffe6ce5e47e8d24b20c6f0597a WatchSource:0}: Error finding container 9031a5946b97c4b0ae20acfff98cf2b58ce51fffe6ce5e47e8d24b20c6f0597a: Status 404 returned error can't find the container with id 9031a5946b97c4b0ae20acfff98cf2b58ce51fffe6ce5e47e8d24b20c6f0597a
Sep 18 13:14:55.665239 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:55.664145    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7608b5b-a581-407b-94b9-3bd63a76c9f0-metrics-certs\") pod \"network-metrics-daemon-h9kt5\" (UID: \"e7608b5b-a581-407b-94b9-3bd63a76c9f0\") " pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:14:55.665239 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:55.664197    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15a0691d-d5b4-41d9-a89c-705ee52f7b25-original-pull-secret\") pod \"global-pull-secret-syncer-bgp65\" (UID: \"15a0691d-d5b4-41d9-a89c-705ee52f7b25\") " pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:14:55.665239 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:55.664259    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqn77\" (UniqueName: \"kubernetes.io/projected/56a3b9a4-38a4-4412-9876-95c3c51c8aff-kube-api-access-lqn77\") pod \"network-check-target-vdj8k\" (UID: \"56a3b9a4-38a4-4412-9876-95c3c51c8aff\") " pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:14:55.665239 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:55.664405    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:14:55.665239 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:55.664423    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:14:55.665239 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:55.664437    2566 projected.go:196] Error preparing data for projected volume kube-api-access-lqn77 for pod openshift-network-diagnostics/network-check-target-vdj8k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:14:55.665239 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:55.664502    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56a3b9a4-38a4-4412-9876-95c3c51c8aff-kube-api-access-lqn77 podName:56a3b9a4-38a4-4412-9876-95c3c51c8aff nodeName:}" failed. No retries permitted until 2026-09-18 13:14:57.664484023 +0000 UTC m=+6.099360431 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lqn77" (UniqueName: "kubernetes.io/projected/56a3b9a4-38a4-4412-9876-95c3c51c8aff-kube-api-access-lqn77") pod "network-check-target-vdj8k" (UID: "56a3b9a4-38a4-4412-9876-95c3c51c8aff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:14:55.665239 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:55.664963    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:14:55.665239 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:55.665018    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7608b5b-a581-407b-94b9-3bd63a76c9f0-metrics-certs podName:e7608b5b-a581-407b-94b9-3bd63a76c9f0 nodeName:}" failed. No retries permitted until 2026-09-18 13:14:57.665001077 +0000 UTC m=+6.099877464 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7608b5b-a581-407b-94b9-3bd63a76c9f0-metrics-certs") pod "network-metrics-daemon-h9kt5" (UID: "e7608b5b-a581-407b-94b9-3bd63a76c9f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:14:55.665239 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:55.665079    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:14:55.665239 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:55.665115    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a0691d-d5b4-41d9-a89c-705ee52f7b25-original-pull-secret podName:15a0691d-d5b4-41d9-a89c-705ee52f7b25 nodeName:}" failed. No retries permitted until 2026-09-18 13:14:57.665102949 +0000 UTC m=+6.099979337 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/15a0691d-d5b4-41d9-a89c-705ee52f7b25-original-pull-secret") pod "global-pull-secret-syncer-bgp65" (UID: "15a0691d-d5b4-41d9-a89c-705ee52f7b25") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:14:56.021849 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:56.021716    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 13:09:54 +0000 UTC" deadline="2028-04-25 01:08:14.320072126 +0000 UTC"
Sep 18 13:14:56.021849 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:56.021759    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14027h53m18.298318274s"
Sep 18 13:14:56.073648 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:56.072859    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:14:56.073648 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:56.072989    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdj8k" podUID="56a3b9a4-38a4-4412-9876-95c3c51c8aff"
Sep 18 13:14:56.102213 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:56.102173    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp" event={"ID":"935f543f-0680-458e-a537-9172ede6c7dd","Type":"ContainerStarted","Data":"182a56b56e4d24e1fe61da243bd5942295be0c6f9fbe7248cf8669287cb11363"}
Sep 18 13:14:56.111688 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:56.111642    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7jvx" event={"ID":"0f82b698-4170-4111-8be8-91ee63365aeb","Type":"ContainerStarted","Data":"6d807b2e2ad81dea4716fb3003f0a3f0d3d100fae2a7cd664b37a44b1a83b222"}
Sep 18 13:14:56.113817 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:56.113787    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b8g55" event={"ID":"df010623-6a98-476b-9bd6-26146ad788c8","Type":"ContainerStarted","Data":"7e4d33eed6d9c4d7e2e40fdd23ce831729bf593ccd0440549bd93cf8b8f94fd3"}
Sep 18 13:14:56.121945 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:56.121196    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-234.ec2.internal" event={"ID":"d6137ead809380c4ef87cd83540755ba","Type":"ContainerStarted","Data":"c6bfd1608b4f858a27f5056dc36a53fa735dd8874b35c2afbfb88e37e2edb427"}
Sep 18 13:14:56.137200 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:56.136955    2566 generic.go:356] "Generic (PLEG): container finished" podID="e90c85b650444ce894d74fab7d1e58d1" containerID="bc1f43b89bf36773f21f72039ce5607c7957009b4112ac4b515464b4e920f8d4" exitCode=0
Sep 18 13:14:56.137200 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:56.137147    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-234.ec2.internal" event={"ID":"e90c85b650444ce894d74fab7d1e58d1","Type":"ContainerDied","Data":"bc1f43b89bf36773f21f72039ce5607c7957009b4112ac4b515464b4e920f8d4"}
Sep 18 13:14:56.145145 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:56.144938    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gcwfs" event={"ID":"c7405cce-1c81-4043-974b-ea69248ca9cb","Type":"ContainerStarted","Data":"fde7d982e7dcae9c92d6443a48c6a24b9c351042bfb5dd6f1e30b96d50954e69"}
Sep 18 13:14:56.159692 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:56.154113    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4" event={"ID":"8836ae4c-f154-4c70-b955-1511e2f6a979","Type":"ContainerStarted","Data":"9031a5946b97c4b0ae20acfff98cf2b58ce51fffe6ce5e47e8d24b20c6f0597a"}
Sep 18 13:14:56.159692 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:56.158926    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4pn75" event={"ID":"06c94dfa-d0af-4756-9cb6-65795af43490","Type":"ContainerStarted","Data":"675acaa277e252a6aa5674756ff1ef4a8a8acc099389ce9902a5d794eb4be5d4"}
Sep 18 13:14:56.172623 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:56.172542    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lglgb" event={"ID":"df2366e7-fd3e-49d3-94af-686a9a9f262f","Type":"ContainerStarted","Data":"3623cfccbd34a0002c4a380021de853f8f885da0e8e0d62c06417935d8540aff"}
Sep 18 13:14:56.175482 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:56.175409    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vrpqk" event={"ID":"8a55b8ec-bd0a-4751-bc08-10e38bf2cc8f","Type":"ContainerStarted","Data":"05aaa4e54d9b03590258a89920f413893c7a85e4ea37040d40a0675147c90dfe"}
Sep 18 13:14:56.178927 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:56.177591    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-234.ec2.internal" podStartSLOduration=2.177574971 podStartE2EDuration="2.177574971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:14:56.141750968 +0000 UTC m=+4.576627379" watchObservedRunningTime="2026-09-18 13:14:56.177574971 +0000 UTC m=+4.612451380"
Sep 18 13:14:56.194344 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:56.194311    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rd9fl" event={"ID":"c6e79ce7-2e57-4103-871f-5b11f0a468d5","Type":"ContainerStarted","Data":"9e3a9ee668cb6edf20e45d024adf4a1a1af0148b009613a0b42985332d3f7176"}
Sep 18 13:14:57.071382 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:57.071342    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:14:57.072028 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:57.071492    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgp65" podUID="15a0691d-d5b4-41d9-a89c-705ee52f7b25"
Sep 18 13:14:57.072028 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:57.071780    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:14:57.072028 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:57.071906    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9kt5" podUID="e7608b5b-a581-407b-94b9-3bd63a76c9f0"
Sep 18 13:14:57.214393 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:57.214304    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-234.ec2.internal" event={"ID":"e90c85b650444ce894d74fab7d1e58d1","Type":"ContainerStarted","Data":"d0fe1e54b5f97c0fe50c96c72df65204aea5f2db596870c0e92ce85e8c3efa3d"}
Sep 18 13:14:57.232442 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:57.231466    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-234.ec2.internal" podStartSLOduration=3.23144203 podStartE2EDuration="3.23144203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:14:57.231091276 +0000 UTC m=+5.665967685" watchObservedRunningTime="2026-09-18 13:14:57.23144203 +0000 UTC m=+5.666318439"
Sep 18 13:14:57.683470 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:57.683373    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqn77\" (UniqueName: \"kubernetes.io/projected/56a3b9a4-38a4-4412-9876-95c3c51c8aff-kube-api-access-lqn77\") pod \"network-check-target-vdj8k\" (UID: \"56a3b9a4-38a4-4412-9876-95c3c51c8aff\") " pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:14:57.683470 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:57.683441    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7608b5b-a581-407b-94b9-3bd63a76c9f0-metrics-certs\") pod \"network-metrics-daemon-h9kt5\" (UID: \"e7608b5b-a581-407b-94b9-3bd63a76c9f0\") " pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:14:57.683724 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:57.683478    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15a0691d-d5b4-41d9-a89c-705ee52f7b25-original-pull-secret\") pod \"global-pull-secret-syncer-bgp65\" (UID: \"15a0691d-d5b4-41d9-a89c-705ee52f7b25\") " pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:14:57.683724 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:57.683605    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:14:57.683724 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:57.683691    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a0691d-d5b4-41d9-a89c-705ee52f7b25-original-pull-secret podName:15a0691d-d5b4-41d9-a89c-705ee52f7b25 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:01.683669392 +0000 UTC m=+10.118545794 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/15a0691d-d5b4-41d9-a89c-705ee52f7b25-original-pull-secret") pod "global-pull-secret-syncer-bgp65" (UID: "15a0691d-d5b4-41d9-a89c-705ee52f7b25") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:14:57.684412 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:57.684032    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:14:57.684412 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:57.684057    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:14:57.684412 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:57.684071    2566 projected.go:196] Error preparing data for projected volume kube-api-access-lqn77 for pod openshift-network-diagnostics/network-check-target-vdj8k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:14:57.684412 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:57.684117    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56a3b9a4-38a4-4412-9876-95c3c51c8aff-kube-api-access-lqn77 podName:56a3b9a4-38a4-4412-9876-95c3c51c8aff nodeName:}" failed. No retries permitted until 2026-09-18 13:15:01.684102174 +0000 UTC m=+10.118978566 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lqn77" (UniqueName: "kubernetes.io/projected/56a3b9a4-38a4-4412-9876-95c3c51c8aff-kube-api-access-lqn77") pod "network-check-target-vdj8k" (UID: "56a3b9a4-38a4-4412-9876-95c3c51c8aff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:14:57.684412 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:57.684182    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:14:57.684412 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:57.684214    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7608b5b-a581-407b-94b9-3bd63a76c9f0-metrics-certs podName:e7608b5b-a581-407b-94b9-3bd63a76c9f0 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:01.68420234 +0000 UTC m=+10.119078730 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7608b5b-a581-407b-94b9-3bd63a76c9f0-metrics-certs") pod "network-metrics-daemon-h9kt5" (UID: "e7608b5b-a581-407b-94b9-3bd63a76c9f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:14:58.074225 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:58.074131    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:14:58.074704 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:58.074275    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdj8k" podUID="56a3b9a4-38a4-4412-9876-95c3c51c8aff"
Sep 18 13:14:59.070734 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:59.070637    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:14:59.070734 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:14:59.070671    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:14:59.070965 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:59.070801    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9kt5" podUID="e7608b5b-a581-407b-94b9-3bd63a76c9f0"
Sep 18 13:14:59.071293 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:14:59.071233    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgp65" podUID="15a0691d-d5b4-41d9-a89c-705ee52f7b25"
Sep 18 13:15:00.073659 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:00.073621    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:15:00.074212 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:00.073792    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdj8k" podUID="56a3b9a4-38a4-4412-9876-95c3c51c8aff"
Sep 18 13:15:01.070590 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:01.070499    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:15:01.070774 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:01.070658    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgp65" podUID="15a0691d-d5b4-41d9-a89c-705ee52f7b25"
Sep 18 13:15:01.071235 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:01.071060    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:15:01.071235 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:01.071185    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9kt5" podUID="e7608b5b-a581-407b-94b9-3bd63a76c9f0"
Sep 18 13:15:01.720065 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:01.720026    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqn77\" (UniqueName: \"kubernetes.io/projected/56a3b9a4-38a4-4412-9876-95c3c51c8aff-kube-api-access-lqn77\") pod \"network-check-target-vdj8k\" (UID: \"56a3b9a4-38a4-4412-9876-95c3c51c8aff\") " pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:15:01.720579 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:01.720188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7608b5b-a581-407b-94b9-3bd63a76c9f0-metrics-certs\") pod \"network-metrics-daemon-h9kt5\" (UID: \"e7608b5b-a581-407b-94b9-3bd63a76c9f0\") " pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:15:01.720579 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:01.720240    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15a0691d-d5b4-41d9-a89c-705ee52f7b25-original-pull-secret\") pod \"global-pull-secret-syncer-bgp65\" (UID: \"15a0691d-d5b4-41d9-a89c-705ee52f7b25\") " pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:15:01.720579 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:01.720243    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:15:01.720579 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:01.720280    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:15:01.720579 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:01.720297    2566 projected.go:196] Error preparing data for projected volume kube-api-access-lqn77 for pod openshift-network-diagnostics/network-check-target-vdj8k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:15:01.720579 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:01.720378    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:15:01.720579 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:01.720494    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:15:01.720579 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:01.720383    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56a3b9a4-38a4-4412-9876-95c3c51c8aff-kube-api-access-lqn77 podName:56a3b9a4-38a4-4412-9876-95c3c51c8aff nodeName:}" failed. No retries permitted until 2026-09-18 13:15:09.720361032 +0000 UTC m=+18.155237419 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lqn77" (UniqueName: "kubernetes.io/projected/56a3b9a4-38a4-4412-9876-95c3c51c8aff-kube-api-access-lqn77") pod "network-check-target-vdj8k" (UID: "56a3b9a4-38a4-4412-9876-95c3c51c8aff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:15:01.720579 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:01.720536    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a0691d-d5b4-41d9-a89c-705ee52f7b25-original-pull-secret podName:15a0691d-d5b4-41d9-a89c-705ee52f7b25 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:09.720523304 +0000 UTC m=+18.155399688 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/15a0691d-d5b4-41d9-a89c-705ee52f7b25-original-pull-secret") pod "global-pull-secret-syncer-bgp65" (UID: "15a0691d-d5b4-41d9-a89c-705ee52f7b25") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:15:01.720579 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:01.720548    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7608b5b-a581-407b-94b9-3bd63a76c9f0-metrics-certs podName:e7608b5b-a581-407b-94b9-3bd63a76c9f0 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:09.720541254 +0000 UTC m=+18.155417637 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7608b5b-a581-407b-94b9-3bd63a76c9f0-metrics-certs") pod "network-metrics-daemon-h9kt5" (UID: "e7608b5b-a581-407b-94b9-3bd63a76c9f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:15:02.072465 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:02.072379    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:15:02.072654 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:02.072505    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdj8k" podUID="56a3b9a4-38a4-4412-9876-95c3c51c8aff"
Sep 18 13:15:03.070987 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:03.070660    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:15:03.070987 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:03.070724    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:15:03.071550 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:03.071099    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9kt5" podUID="e7608b5b-a581-407b-94b9-3bd63a76c9f0"
Sep 18 13:15:03.071610 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:03.071559    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgp65" podUID="15a0691d-d5b4-41d9-a89c-705ee52f7b25"
Sep 18 13:15:04.070915 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:04.070800    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:15:04.071070 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:04.070925    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdj8k" podUID="56a3b9a4-38a4-4412-9876-95c3c51c8aff"
Sep 18 13:15:05.070471 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:05.070440    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:15:05.070633 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:05.070435    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:15:05.070633 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:05.070575    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9kt5" podUID="e7608b5b-a581-407b-94b9-3bd63a76c9f0"
Sep 18 13:15:05.070633 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:05.070620    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgp65" podUID="15a0691d-d5b4-41d9-a89c-705ee52f7b25"
Sep 18 13:15:06.071315 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:06.071293    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:15:06.071585 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:06.071392    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdj8k" podUID="56a3b9a4-38a4-4412-9876-95c3c51c8aff"
Sep 18 13:15:07.070672 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:07.070635    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:15:07.070858 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:07.070682    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:15:07.070858 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:07.070761    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgp65" podUID="15a0691d-d5b4-41d9-a89c-705ee52f7b25"
Sep 18 13:15:07.070963 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:07.070903    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9kt5" podUID="e7608b5b-a581-407b-94b9-3bd63a76c9f0"
Sep 18 13:15:08.070699 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:08.070632    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:15:08.071159 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:08.070769    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdj8k" podUID="56a3b9a4-38a4-4412-9876-95c3c51c8aff"
Sep 18 13:15:09.071418 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:09.071372    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:15:09.072009 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:09.071506    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgp65" podUID="15a0691d-d5b4-41d9-a89c-705ee52f7b25"
Sep 18 13:15:09.072009 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:09.071731    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:15:09.072009 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:09.071834    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9kt5" podUID="e7608b5b-a581-407b-94b9-3bd63a76c9f0"
Sep 18 13:15:09.777757 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:09.777716    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqn77\" (UniqueName: \"kubernetes.io/projected/56a3b9a4-38a4-4412-9876-95c3c51c8aff-kube-api-access-lqn77\") pod \"network-check-target-vdj8k\" (UID: \"56a3b9a4-38a4-4412-9876-95c3c51c8aff\") " pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:15:09.777956 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:09.777778    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7608b5b-a581-407b-94b9-3bd63a76c9f0-metrics-certs\") pod \"network-metrics-daemon-h9kt5\" (UID: \"e7608b5b-a581-407b-94b9-3bd63a76c9f0\") " pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:15:09.777956 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:09.777811    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15a0691d-d5b4-41d9-a89c-705ee52f7b25-original-pull-secret\") pod \"global-pull-secret-syncer-bgp65\" (UID: \"15a0691d-d5b4-41d9-a89c-705ee52f7b25\") " pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:15:09.777956 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:09.777913    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:15:09.777956 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:09.777954    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:15:09.778161 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:09.777971    2566 projected.go:196] Error preparing data for projected volume kube-api-access-lqn77 for pod openshift-network-diagnostics/network-check-target-vdj8k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:15:09.778161 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:09.777913    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:15:09.778161 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:09.778020    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:15:09.778161 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:09.778037    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56a3b9a4-38a4-4412-9876-95c3c51c8aff-kube-api-access-lqn77 podName:56a3b9a4-38a4-4412-9876-95c3c51c8aff nodeName:}" failed. No retries permitted until 2026-09-18 13:15:25.778023103 +0000 UTC m=+34.212899503 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lqn77" (UniqueName: "kubernetes.io/projected/56a3b9a4-38a4-4412-9876-95c3c51c8aff-kube-api-access-lqn77") pod "network-check-target-vdj8k" (UID: "56a3b9a4-38a4-4412-9876-95c3c51c8aff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:15:09.778161 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:09.778060    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7608b5b-a581-407b-94b9-3bd63a76c9f0-metrics-certs podName:e7608b5b-a581-407b-94b9-3bd63a76c9f0 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:25.778049141 +0000 UTC m=+34.212925522 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7608b5b-a581-407b-94b9-3bd63a76c9f0-metrics-certs") pod "network-metrics-daemon-h9kt5" (UID: "e7608b5b-a581-407b-94b9-3bd63a76c9f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:15:09.778161 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:09.778075    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a0691d-d5b4-41d9-a89c-705ee52f7b25-original-pull-secret podName:15a0691d-d5b4-41d9-a89c-705ee52f7b25 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:25.77806681 +0000 UTC m=+34.212943192 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/15a0691d-d5b4-41d9-a89c-705ee52f7b25-original-pull-secret") pod "global-pull-secret-syncer-bgp65" (UID: "15a0691d-d5b4-41d9-a89c-705ee52f7b25") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:15:10.070858 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:10.070775    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:15:10.071061 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:10.070911    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdj8k" podUID="56a3b9a4-38a4-4412-9876-95c3c51c8aff"
Sep 18 13:15:11.070451 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:11.070415    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:15:11.070953 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:11.070415    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:15:11.070953 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:11.070551    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgp65" podUID="15a0691d-d5b4-41d9-a89c-705ee52f7b25"
Sep 18 13:15:11.070953 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:11.070663    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9kt5" podUID="e7608b5b-a581-407b-94b9-3bd63a76c9f0"
Sep 18 13:15:12.071728 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:12.071690    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:15:12.072213 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:12.071799    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdj8k" podUID="56a3b9a4-38a4-4412-9876-95c3c51c8aff"
Sep 18 13:15:13.070757 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:13.070722    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:15:13.070942 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:13.070728    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:15:13.070942 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:13.070860    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgp65" podUID="15a0691d-d5b4-41d9-a89c-705ee52f7b25"
Sep 18 13:15:13.070942 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:13.070930    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9kt5" podUID="e7608b5b-a581-407b-94b9-3bd63a76c9f0"
Sep 18 13:15:14.071313 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:14.071109    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:15:14.072183 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:14.071390    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdj8k" podUID="56a3b9a4-38a4-4412-9876-95c3c51c8aff"
Sep 18 13:15:14.251914 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:14.251837    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rd9fl" event={"ID":"c6e79ce7-2e57-4103-871f-5b11f0a468d5","Type":"ContainerStarted","Data":"44786654e74f9eddb12fbce2c76aaa978eba51909575f329235bda5314ff1733"}
Sep 18 13:15:14.253643 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:14.253620    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp" event={"ID":"935f543f-0680-458e-a537-9172ede6c7dd","Type":"ContainerStarted","Data":"006735a84cd547f9b83310d45a1b1d72f7b720b5c8bc76e460a31e364e9f1673"}
Sep 18 13:15:14.255471 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:14.255446    2566 generic.go:356] "Generic (PLEG): container finished" podID="0f82b698-4170-4111-8be8-91ee63365aeb" containerID="3fb0692edb92596ed135d7c2ea51d0c859a85b067024f455179c0a2e891268da" exitCode=0
Sep 18 13:15:14.255573 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:14.255512    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7jvx" event={"ID":"0f82b698-4170-4111-8be8-91ee63365aeb","Type":"ContainerDied","Data":"3fb0692edb92596ed135d7c2ea51d0c859a85b067024f455179c0a2e891268da"}
Sep 18 13:15:14.259027 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:14.259001    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b8g55" event={"ID":"df010623-6a98-476b-9bd6-26146ad788c8","Type":"ContainerStarted","Data":"d6f18514140f013b87dc93c2e849d2fe18c9a002dfc1c73639b6c595bd06fcb9"}
Sep 18 13:15:14.260777 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:14.260747    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gcwfs" event={"ID":"c7405cce-1c81-4043-974b-ea69248ca9cb","Type":"ContainerStarted","Data":"fd0843940bc2d38718ed5ac1afea69e84cacb119560c3a276aeadd08b3bda979"}
Sep 18 13:15:14.264114 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:14.264093    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4" event={"ID":"8836ae4c-f154-4c70-b955-1511e2f6a979","Type":"ContainerStarted","Data":"e775885c7b37c2f2d213ab1f314cd14fc208f5bb64843c9a553d5188b58f1fa5"}
Sep 18 13:15:14.264214 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:14.264167    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4" event={"ID":"8836ae4c-f154-4c70-b955-1511e2f6a979","Type":"ContainerStarted","Data":"b1947b168f04881ad8c1fbc22336c458f046432e96e50c7bb2b28e831ab513b9"}
Sep 18 13:15:14.264260 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:14.264215    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4" event={"ID":"8836ae4c-f154-4c70-b955-1511e2f6a979","Type":"ContainerStarted","Data":"8a0ab24776dff0cf063e32cbb64f914bf6c49d7f5c7fedd4518793134327e32d"}
Sep 18 13:15:14.264314 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:14.264261    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4" event={"ID":"8836ae4c-f154-4c70-b955-1511e2f6a979","Type":"ContainerStarted","Data":"3f279b85141ac69d72dd567b74e351a27e795b7d1bbc0ec5e9178de288007778"}
Sep 18 13:15:14.264314 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:14.264310    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4" event={"ID":"8836ae4c-f154-4c70-b955-1511e2f6a979","Type":"ContainerStarted","Data":"1cf01b0590063e9dca1f21173562755f75b43f470a3841f2334981070be6626d"}
Sep 18 13:15:14.264415 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:14.264356    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4" event={"ID":"8836ae4c-f154-4c70-b955-1511e2f6a979","Type":"ContainerStarted","Data":"546ce5be1d36c9db1425834bb7639df8e8f0294911f046ad2c4a0808321de78e"}
Sep 18 13:15:14.265691 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:14.265662    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4pn75" event={"ID":"06c94dfa-d0af-4756-9cb6-65795af43490","Type":"ContainerStarted","Data":"3c9b1a265e059ab9a9b3a421f83f7105595b5225ec0752f7d7597fc4c0a86e2e"}
Sep 18 13:15:14.267554 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:14.267343    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lglgb" event={"ID":"df2366e7-fd3e-49d3-94af-686a9a9f262f","Type":"ContainerStarted","Data":"77ff571933a7ab6db8fa7d82c13764d70c18092a8c5e54ea51f503915939bf82"}
Sep 18 13:15:14.267895 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:14.267856    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rd9fl" podStartSLOduration=3.061222768 podStartE2EDuration="21.267842168s" totalImagesPullingTime="18.2066194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:14.267405012 +0000 UTC m=+22.702281420" watchObservedRunningTime="2026-09-18 13:15:14.267842168 +0000 UTC m=+22.702718573"
Sep 18 13:15:14.297331 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:14.297284    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-b8g55" podStartSLOduration=11.291861663 podStartE2EDuration="22.297264252s" totalImagesPullingTime="11.005402589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:14.28148231 +0000 UTC m=+22.716358714" watchObservedRunningTime="2026-09-18 13:15:14.297264252 +0000 UTC m=+22.732140656"
Sep 18 13:15:14.297468 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:14.297386    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gcwfs" podStartSLOduration=4.102957547 podStartE2EDuration="22.297379302s" totalImagesPullingTime="18.194421755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:14.296830631 +0000 UTC m=+22.731707036" watchObservedRunningTime="2026-09-18 13:15:14.297379302 +0000 UTC m=+22.732255707"
Sep 18 13:15:14.314691 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:14.314651    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4pn75" podStartSLOduration=3.059357282 podStartE2EDuration="21.314636775s" totalImagesPullingTime="18.255279493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:14.314511335 +0000 UTC m=+22.749387741" watchObservedRunningTime="2026-09-18 13:15:14.314636775 +0000 UTC m=+22.749513192"
Sep 18 13:15:14.332447 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:14.332402    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lglgb" podStartSLOduration=3.131653354 podStartE2EDuration="21.332389887s" totalImagesPullingTime="18.200736533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:14.331944039 +0000 UTC m=+22.766820443" watchObservedRunningTime="2026-09-18 13:15:14.332389887 +0000 UTC m=+22.767266292"
Sep 18 13:15:14.698350 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:14.698195    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 13:15:14.993784 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:14.993606    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T13:15:14.69834309Z","UUID":"481988d5-f508-4843-accb-44366f01fae9","Handler":null,"Name":"","Endpoint":""}
Sep 18 13:15:14.996556 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:14.996534    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 13:15:14.996556 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:14.996569    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 13:15:15.070679 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:15.070646    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:15:15.070858 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:15.070646    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:15:15.070858 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:15.070791    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9kt5" podUID="e7608b5b-a581-407b-94b9-3bd63a76c9f0"
Sep 18 13:15:15.070858 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:15.070844    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgp65" podUID="15a0691d-d5b4-41d9-a89c-705ee52f7b25"
Sep 18 13:15:15.272745 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:15.272629    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vrpqk" event={"ID":"8a55b8ec-bd0a-4751-bc08-10e38bf2cc8f","Type":"ContainerStarted","Data":"2cb2799f02a52c99263c915ee9c3d9a56480d34f8145d1960b699e51bf2fc278"}
Sep 18 13:15:15.275916 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:15.275088    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp" event={"ID":"935f543f-0680-458e-a537-9172ede6c7dd","Type":"ContainerStarted","Data":"17ed8c6ec1e8a80752b07b49d5c426b982e759a254511382705df10d3dd2ec05"}
Sep 18 13:15:15.287379 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:15.287332    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vrpqk" podStartSLOduration=4.092094306 podStartE2EDuration="22.287316612s" totalImagesPullingTime="18.195222306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:15.287095931 +0000 UTC m=+23.721972336" watchObservedRunningTime="2026-09-18 13:15:15.287316612 +0000 UTC m=+23.722193019"
Sep 18 13:15:16.071425 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:16.071394    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:15:16.071612 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:16.071519    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdj8k" podUID="56a3b9a4-38a4-4412-9876-95c3c51c8aff"
Sep 18 13:15:16.281264 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:16.281179    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4" event={"ID":"8836ae4c-f154-4c70-b955-1511e2f6a979","Type":"ContainerStarted","Data":"dd6f423a91fa58133acc1934651fbd22d4cc6b13b4a4196ae8f7cade57c5d57f"}
Sep 18 13:15:16.283550 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:16.283523    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp" event={"ID":"935f543f-0680-458e-a537-9172ede6c7dd","Type":"ContainerStarted","Data":"75757e3b1f13eb007df4d1af18286e8ad6729b1e695737d533df7ec1ded0e528"}
Sep 18 13:15:16.318922 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:16.318874    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z94bp" podStartSLOduration=4.107956405 podStartE2EDuration="24.31885922s" totalImagesPullingTime="20.210902815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:14:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:16.318622272 +0000 UTC m=+24.753498674" watchObservedRunningTime="2026-09-18 13:15:16.31885922 +0000 UTC m=+24.753735624"
Sep 18 13:15:17.071283 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:17.071254    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:15:17.071488 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:17.071254    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:15:17.071488 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:17.071379    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9kt5" podUID="e7608b5b-a581-407b-94b9-3bd63a76c9f0"
Sep 18 13:15:17.071488 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:17.071446    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgp65" podUID="15a0691d-d5b4-41d9-a89c-705ee52f7b25"
Sep 18 13:15:18.071196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:18.071170    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:15:18.071578 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:18.071279    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdj8k" podUID="56a3b9a4-38a4-4412-9876-95c3c51c8aff"
Sep 18 13:15:18.540333 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:18.540112    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-b8g55"
Sep 18 13:15:18.541045 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:18.540865    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-b8g55"
Sep 18 13:15:19.070347 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:19.070324    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:15:19.070491 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:19.070324    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:15:19.070491 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:19.070430    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9kt5" podUID="e7608b5b-a581-407b-94b9-3bd63a76c9f0"
Sep 18 13:15:19.070577 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:19.070529    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgp65" podUID="15a0691d-d5b4-41d9-a89c-705ee52f7b25"
Sep 18 13:15:19.293516 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:19.293480    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4" event={"ID":"8836ae4c-f154-4c70-b955-1511e2f6a979","Type":"ContainerStarted","Data":"5598cb2f468486d4c12baa38cde5853ed8662a1663590971edea0c8e8e329357"}
Sep 18 13:15:19.294088 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:19.293751    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:15:19.298200 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:19.298172    2566 generic.go:356] "Generic (PLEG): container finished" podID="0f82b698-4170-4111-8be8-91ee63365aeb" containerID="8e3e2358a1ad9d7065736d302d108d36d13bb8c8994798bebca331fd6a656e95" exitCode=0
Sep 18 13:15:19.298328 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:19.298247    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7jvx" event={"ID":"0f82b698-4170-4111-8be8-91ee63365aeb","Type":"ContainerDied","Data":"8e3e2358a1ad9d7065736d302d108d36d13bb8c8994798bebca331fd6a656e95"}
Sep 18 13:15:19.298470 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:19.298442    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b8g55"
Sep 18 13:15:19.299390 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:19.298973    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b8g55"
Sep 18 13:15:19.311753 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:19.311735    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:15:19.335612 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:19.335529    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4" podStartSLOduration=8.101465022 podStartE2EDuration="26.335518137s" totalImagesPullingTime="18.234053115s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:19.335450581 +0000 UTC m=+27.770326986" watchObservedRunningTime="2026-09-18 13:15:19.335518137 +0000 UTC m=+27.770394540"
Sep 18 13:15:20.071038 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:20.071006    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:15:20.071239 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:20.071118    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdj8k" podUID="56a3b9a4-38a4-4412-9876-95c3c51c8aff"
Sep 18 13:15:20.302501 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:20.302469    2566 generic.go:356] "Generic (PLEG): container finished" podID="0f82b698-4170-4111-8be8-91ee63365aeb" containerID="8d9500523ba8d9e2731ef12055e947bfb1999b4976e493ffe04d2cd87db5b196" exitCode=0
Sep 18 13:15:20.303264 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:20.302546    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7jvx" event={"ID":"0f82b698-4170-4111-8be8-91ee63365aeb","Type":"ContainerDied","Data":"8d9500523ba8d9e2731ef12055e947bfb1999b4976e493ffe04d2cd87db5b196"}
Sep 18 13:15:20.303264 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:20.303061    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:15:20.303264 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:20.303139    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:15:20.318211 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:20.318187    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:15:20.635548 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:20.635364    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bgp65"]
Sep 18 13:15:20.635734 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:20.635685    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:15:20.635792 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:20.635770    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgp65" podUID="15a0691d-d5b4-41d9-a89c-705ee52f7b25"
Sep 18 13:15:20.635877 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:20.635857    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h9kt5"]
Sep 18 13:15:20.635982 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:20.635970    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:15:20.636073 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:20.636054    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9kt5" podUID="e7608b5b-a581-407b-94b9-3bd63a76c9f0"
Sep 18 13:15:20.637158 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:20.637136    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vdj8k"]
Sep 18 13:15:20.637296 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:20.637229    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:15:20.637363 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:20.637317    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdj8k" podUID="56a3b9a4-38a4-4412-9876-95c3c51c8aff"
Sep 18 13:15:21.307861 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:21.307777    2566 generic.go:356] "Generic (PLEG): container finished" podID="0f82b698-4170-4111-8be8-91ee63365aeb" containerID="4e9a5b339a76737abee1e47d7870f5f083902dfb7bbbe83239555532ee7e60c0" exitCode=0
Sep 18 13:15:21.308288 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:21.307862    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7jvx" event={"ID":"0f82b698-4170-4111-8be8-91ee63365aeb","Type":"ContainerDied","Data":"4e9a5b339a76737abee1e47d7870f5f083902dfb7bbbe83239555532ee7e60c0"}
Sep 18 13:15:22.071491 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:22.071459    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:15:22.071797 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:22.071539    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:15:22.071919 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:22.071814    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdj8k" podUID="56a3b9a4-38a4-4412-9876-95c3c51c8aff"
Sep 18 13:15:22.071989 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:22.071919    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9kt5" podUID="e7608b5b-a581-407b-94b9-3bd63a76c9f0"
Sep 18 13:15:22.071989 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:22.071562    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:15:22.072098 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:22.072030    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgp65" podUID="15a0691d-d5b4-41d9-a89c-705ee52f7b25"
Sep 18 13:15:24.071445 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:24.071360    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:15:24.071445 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:24.071357    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:15:24.071979 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:24.071500    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:15:24.071979 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:24.071502    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgp65" podUID="15a0691d-d5b4-41d9-a89c-705ee52f7b25"
Sep 18 13:15:24.071979 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:24.071639    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdj8k" podUID="56a3b9a4-38a4-4412-9876-95c3c51c8aff"
Sep 18 13:15:24.071979 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:24.071726    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9kt5" podUID="e7608b5b-a581-407b-94b9-3bd63a76c9f0"
Sep 18 13:15:25.791243 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:25.791156    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7608b5b-a581-407b-94b9-3bd63a76c9f0-metrics-certs\") pod \"network-metrics-daemon-h9kt5\" (UID: \"e7608b5b-a581-407b-94b9-3bd63a76c9f0\") " pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:15:25.791243 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:25.791202    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15a0691d-d5b4-41d9-a89c-705ee52f7b25-original-pull-secret\") pod \"global-pull-secret-syncer-bgp65\" (UID: \"15a0691d-d5b4-41d9-a89c-705ee52f7b25\") " pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:15:25.791777 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:25.791252    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqn77\" (UniqueName: \"kubernetes.io/projected/56a3b9a4-38a4-4412-9876-95c3c51c8aff-kube-api-access-lqn77\") pod \"network-check-target-vdj8k\" (UID: \"56a3b9a4-38a4-4412-9876-95c3c51c8aff\") " pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:15:25.791777 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:25.791310    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:15:25.791777 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:25.791369    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:15:25.791777 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:25.791388    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7608b5b-a581-407b-94b9-3bd63a76c9f0-metrics-certs podName:e7608b5b-a581-407b-94b9-3bd63a76c9f0 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:57.791357691 +0000 UTC m=+66.226234077 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7608b5b-a581-407b-94b9-3bd63a76c9f0-metrics-certs") pod "network-metrics-daemon-h9kt5" (UID: "e7608b5b-a581-407b-94b9-3bd63a76c9f0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:15:25.791777 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:25.791375    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:15:25.791777 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:25.791452    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:15:25.791777 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:25.791422    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a0691d-d5b4-41d9-a89c-705ee52f7b25-original-pull-secret podName:15a0691d-d5b4-41d9-a89c-705ee52f7b25 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:57.791409922 +0000 UTC m=+66.226286304 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/15a0691d-d5b4-41d9-a89c-705ee52f7b25-original-pull-secret") pod "global-pull-secret-syncer-bgp65" (UID: "15a0691d-d5b4-41d9-a89c-705ee52f7b25") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:15:25.791777 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:25.791467    2566 projected.go:196] Error preparing data for projected volume kube-api-access-lqn77 for pod openshift-network-diagnostics/network-check-target-vdj8k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:15:25.791777 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:25.791503    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56a3b9a4-38a4-4412-9876-95c3c51c8aff-kube-api-access-lqn77 podName:56a3b9a4-38a4-4412-9876-95c3c51c8aff nodeName:}" failed. No retries permitted until 2026-09-18 13:15:57.79149206 +0000 UTC m=+66.226368446 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lqn77" (UniqueName: "kubernetes.io/projected/56a3b9a4-38a4-4412-9876-95c3c51c8aff-kube-api-access-lqn77") pod "network-check-target-vdj8k" (UID: "56a3b9a4-38a4-4412-9876-95c3c51c8aff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:15:26.071334 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.071247    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:15:26.071611 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.071247    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:15:26.071611 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:26.071362    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h9kt5" podUID="e7608b5b-a581-407b-94b9-3bd63a76c9f0"
Sep 18 13:15:26.071611 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.071397    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:15:26.071611 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:26.071479    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vdj8k" podUID="56a3b9a4-38a4-4412-9876-95c3c51c8aff"
Sep 18 13:15:26.071611 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:26.071555    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bgp65" podUID="15a0691d-d5b4-41d9-a89c-705ee52f7b25"
Sep 18 13:15:26.321402 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.321320    2566 generic.go:356] "Generic (PLEG): container finished" podID="0f82b698-4170-4111-8be8-91ee63365aeb" containerID="7e183714d09adc0caea9794841dafedd7d95b3157004d5cb31b2e6434f2042e3" exitCode=0
Sep 18 13:15:26.321402 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.321363    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7jvx" event={"ID":"0f82b698-4170-4111-8be8-91ee63365aeb","Type":"ContainerDied","Data":"7e183714d09adc0caea9794841dafedd7d95b3157004d5cb31b2e6434f2042e3"}
Sep 18 13:15:26.329234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.329205    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-234.ec2.internal" event="NodeReady"
Sep 18 13:15:26.329361 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.329346    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 13:15:26.447001 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.446975    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r94bv"]
Sep 18 13:15:26.469578 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.469552    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cn6sm"]
Sep 18 13:15:26.469692 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.469675    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r94bv"
Sep 18 13:15:26.483778 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.483751    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cn6sm"
Sep 18 13:15:26.501685 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.501662    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 13:15:26.501812 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.501664    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 13:15:26.501812 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.501665    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5qlvl\""
Sep 18 13:15:26.501961 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.501944    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 13:15:26.502025 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.501991    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 13:15:26.502180 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.502164    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 13:15:26.507168 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.507152    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8m5tk\""
Sep 18 13:15:26.537328 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.537298    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cn6sm"]
Sep 18 13:15:26.538145 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.538122    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r94bv"]
Sep 18 13:15:26.597125 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.597095    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzqkr\" (UniqueName: \"kubernetes.io/projected/96cd73fd-fcb1-44ea-b184-63a97a0277f4-kube-api-access-jzqkr\") pod \"dns-default-r94bv\" (UID: \"96cd73fd-fcb1-44ea-b184-63a97a0277f4\") " pod="openshift-dns/dns-default-r94bv"
Sep 18 13:15:26.597307 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.597135    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96cd73fd-fcb1-44ea-b184-63a97a0277f4-tmp-dir\") pod \"dns-default-r94bv\" (UID: \"96cd73fd-fcb1-44ea-b184-63a97a0277f4\") " pod="openshift-dns/dns-default-r94bv"
Sep 18 13:15:26.597307 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.597203    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/96cd73fd-fcb1-44ea-b184-63a97a0277f4-config-volume\") pod \"dns-default-r94bv\" (UID: \"96cd73fd-fcb1-44ea-b184-63a97a0277f4\") " pod="openshift-dns/dns-default-r94bv"
Sep 18 13:15:26.597307 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.597239    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2452828a-bce2-4da3-a477-88d9d048a4b3-cert\") pod \"ingress-canary-cn6sm\" (UID: \"2452828a-bce2-4da3-a477-88d9d048a4b3\") " pod="openshift-ingress-canary/ingress-canary-cn6sm"
Sep 18 13:15:26.597307 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.597277    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96cd73fd-fcb1-44ea-b184-63a97a0277f4-metrics-tls\") pod \"dns-default-r94bv\" (UID: \"96cd73fd-fcb1-44ea-b184-63a97a0277f4\") " pod="openshift-dns/dns-default-r94bv"
Sep 18 13:15:26.597307 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.597303    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvq5d\" (UniqueName: \"kubernetes.io/projected/2452828a-bce2-4da3-a477-88d9d048a4b3-kube-api-access-tvq5d\") pod \"ingress-canary-cn6sm\" (UID: \"2452828a-bce2-4da3-a477-88d9d048a4b3\") " pod="openshift-ingress-canary/ingress-canary-cn6sm"
Sep 18 13:15:26.698755 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.698560    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/96cd73fd-fcb1-44ea-b184-63a97a0277f4-config-volume\") pod \"dns-default-r94bv\" (UID: \"96cd73fd-fcb1-44ea-b184-63a97a0277f4\") " pod="openshift-dns/dns-default-r94bv"
Sep 18 13:15:26.698906 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.698774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2452828a-bce2-4da3-a477-88d9d048a4b3-cert\") pod \"ingress-canary-cn6sm\" (UID: \"2452828a-bce2-4da3-a477-88d9d048a4b3\") " pod="openshift-ingress-canary/ingress-canary-cn6sm"
Sep 18 13:15:26.698906 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.698820    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96cd73fd-fcb1-44ea-b184-63a97a0277f4-metrics-tls\") pod \"dns-default-r94bv\" (UID: \"96cd73fd-fcb1-44ea-b184-63a97a0277f4\") " pod="openshift-dns/dns-default-r94bv"
Sep 18 13:15:26.698906 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.698844    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvq5d\" (UniqueName: \"kubernetes.io/projected/2452828a-bce2-4da3-a477-88d9d048a4b3-kube-api-access-tvq5d\") pod \"ingress-canary-cn6sm\" (UID: \"2452828a-bce2-4da3-a477-88d9d048a4b3\") " pod="openshift-ingress-canary/ingress-canary-cn6sm"
Sep 18 13:15:26.698906 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.698881    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzqkr\" (UniqueName: \"kubernetes.io/projected/96cd73fd-fcb1-44ea-b184-63a97a0277f4-kube-api-access-jzqkr\") pod \"dns-default-r94bv\" (UID: \"96cd73fd-fcb1-44ea-b184-63a97a0277f4\") " pod="openshift-dns/dns-default-r94bv"
Sep 18 13:15:26.699117 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.698909    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96cd73fd-fcb1-44ea-b184-63a97a0277f4-tmp-dir\") pod \"dns-default-r94bv\" (UID: \"96cd73fd-fcb1-44ea-b184-63a97a0277f4\") " pod="openshift-dns/dns-default-r94bv"
Sep 18 13:15:26.699117 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:26.698961    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:15:26.699117 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:26.699030    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96cd73fd-fcb1-44ea-b184-63a97a0277f4-metrics-tls podName:96cd73fd-fcb1-44ea-b184-63a97a0277f4 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:27.199009548 +0000 UTC m=+35.633885954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96cd73fd-fcb1-44ea-b184-63a97a0277f4-metrics-tls") pod "dns-default-r94bv" (UID: "96cd73fd-fcb1-44ea-b184-63a97a0277f4") : secret "dns-default-metrics-tls" not found
Sep 18 13:15:26.699253 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.699200    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96cd73fd-fcb1-44ea-b184-63a97a0277f4-tmp-dir\") pod \"dns-default-r94bv\" (UID: \"96cd73fd-fcb1-44ea-b184-63a97a0277f4\") " pod="openshift-dns/dns-default-r94bv"
Sep 18 13:15:26.699340 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.699321    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/96cd73fd-fcb1-44ea-b184-63a97a0277f4-config-volume\") pod \"dns-default-r94bv\" (UID: \"96cd73fd-fcb1-44ea-b184-63a97a0277f4\") " pod="openshift-dns/dns-default-r94bv"
Sep 18 13:15:26.702989 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.702964    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2452828a-bce2-4da3-a477-88d9d048a4b3-cert\") pod \"ingress-canary-cn6sm\" (UID: \"2452828a-bce2-4da3-a477-88d9d048a4b3\") " pod="openshift-ingress-canary/ingress-canary-cn6sm"
Sep 18 13:15:26.713936 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.713910    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvq5d\" (UniqueName: \"kubernetes.io/projected/2452828a-bce2-4da3-a477-88d9d048a4b3-kube-api-access-tvq5d\") pod \"ingress-canary-cn6sm\" (UID: \"2452828a-bce2-4da3-a477-88d9d048a4b3\") " pod="openshift-ingress-canary/ingress-canary-cn6sm"
Sep 18 13:15:26.714814 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.714793    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzqkr\" (UniqueName: \"kubernetes.io/projected/96cd73fd-fcb1-44ea-b184-63a97a0277f4-kube-api-access-jzqkr\") pod \"dns-default-r94bv\" (UID: \"96cd73fd-fcb1-44ea-b184-63a97a0277f4\") " pod="openshift-dns/dns-default-r94bv"
Sep 18 13:15:26.792446 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:26.792401    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cn6sm"
Sep 18 13:15:27.119652 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:27.119619    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cn6sm"]
Sep 18 13:15:27.123345 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:15:27.123321    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2452828a_bce2_4da3_a477_88d9d048a4b3.slice/crio-ea17c741d063b1844b8cc9a3a1ac742f780f7db15300c8d5edaa31d2c2f9f99e WatchSource:0}: Error finding container ea17c741d063b1844b8cc9a3a1ac742f780f7db15300c8d5edaa31d2c2f9f99e: Status 404 returned error can't find the container with id ea17c741d063b1844b8cc9a3a1ac742f780f7db15300c8d5edaa31d2c2f9f99e
Sep 18 13:15:27.202151 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:27.202058    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96cd73fd-fcb1-44ea-b184-63a97a0277f4-metrics-tls\") pod \"dns-default-r94bv\" (UID: \"96cd73fd-fcb1-44ea-b184-63a97a0277f4\") " pod="openshift-dns/dns-default-r94bv"
Sep 18 13:15:27.202301 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:27.202208    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:15:27.202301 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:27.202273    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96cd73fd-fcb1-44ea-b184-63a97a0277f4-metrics-tls podName:96cd73fd-fcb1-44ea-b184-63a97a0277f4 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:28.202258326 +0000 UTC m=+36.637134708 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96cd73fd-fcb1-44ea-b184-63a97a0277f4-metrics-tls") pod "dns-default-r94bv" (UID: "96cd73fd-fcb1-44ea-b184-63a97a0277f4") : secret "dns-default-metrics-tls" not found
Sep 18 13:15:27.327210 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:27.327170    2566 generic.go:356] "Generic (PLEG): container finished" podID="0f82b698-4170-4111-8be8-91ee63365aeb" containerID="5740926493fe9c47d50e908a2166acfddb5b3aadea83fd1e5b3a459de5d61335" exitCode=0
Sep 18 13:15:27.327387 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:27.327229    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7jvx" event={"ID":"0f82b698-4170-4111-8be8-91ee63365aeb","Type":"ContainerDied","Data":"5740926493fe9c47d50e908a2166acfddb5b3aadea83fd1e5b3a459de5d61335"}
Sep 18 13:15:27.328442 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:27.328410    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cn6sm" event={"ID":"2452828a-bce2-4da3-a477-88d9d048a4b3","Type":"ContainerStarted","Data":"ea17c741d063b1844b8cc9a3a1ac742f780f7db15300c8d5edaa31d2c2f9f99e"}
Sep 18 13:15:28.070447 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:28.070412    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:15:28.071303 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:28.070409    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:15:28.071303 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:28.070412    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:15:28.075878 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:28.075853    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 13:15:28.077097 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:28.077077    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 13:15:28.077240 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:28.077115    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 13:15:28.077240 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:28.077180    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4rmbt\""
Sep 18 13:15:28.077240 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:28.077112    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 13:15:28.077240 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:28.077213    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-crtpr\""
Sep 18 13:15:28.213080 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:28.213045    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96cd73fd-fcb1-44ea-b184-63a97a0277f4-metrics-tls\") pod \"dns-default-r94bv\" (UID: \"96cd73fd-fcb1-44ea-b184-63a97a0277f4\") " pod="openshift-dns/dns-default-r94bv"
Sep 18 13:15:28.213264 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:28.213208    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:15:28.213318 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:28.213279    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96cd73fd-fcb1-44ea-b184-63a97a0277f4-metrics-tls podName:96cd73fd-fcb1-44ea-b184-63a97a0277f4 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:30.213258559 +0000 UTC m=+38.648134964 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96cd73fd-fcb1-44ea-b184-63a97a0277f4-metrics-tls") pod "dns-default-r94bv" (UID: "96cd73fd-fcb1-44ea-b184-63a97a0277f4") : secret "dns-default-metrics-tls" not found
Sep 18 13:15:28.336303 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:28.336222    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t7jvx" event={"ID":"0f82b698-4170-4111-8be8-91ee63365aeb","Type":"ContainerStarted","Data":"59610e84aef4db946baa53d5ab21b4c5a456d54b54ad7dba881f09ee3e008fdf"}
Sep 18 13:15:28.375048 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:28.374987    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t7jvx" podStartSLOduration=7.334710286 podStartE2EDuration="35.374969061s" totalImagesPullingTime="28.040258775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:14:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:28.374352425 +0000 UTC m=+36.809228833" watchObservedRunningTime="2026-09-18 13:15:28.374969061 +0000 UTC m=+36.809845466"
Sep 18 13:15:30.231638 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:30.231385    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96cd73fd-fcb1-44ea-b184-63a97a0277f4-metrics-tls\") pod \"dns-default-r94bv\" (UID: \"96cd73fd-fcb1-44ea-b184-63a97a0277f4\") " pod="openshift-dns/dns-default-r94bv"
Sep 18 13:15:30.231638 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:30.231542    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:15:30.232028 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:30.231685    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96cd73fd-fcb1-44ea-b184-63a97a0277f4-metrics-tls podName:96cd73fd-fcb1-44ea-b184-63a97a0277f4 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:34.23167135 +0000 UTC m=+42.666547737 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96cd73fd-fcb1-44ea-b184-63a97a0277f4-metrics-tls") pod "dns-default-r94bv" (UID: "96cd73fd-fcb1-44ea-b184-63a97a0277f4") : secret "dns-default-metrics-tls" not found
Sep 18 13:15:30.345294 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:30.345263    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cn6sm" event={"ID":"2452828a-bce2-4da3-a477-88d9d048a4b3","Type":"ContainerStarted","Data":"e0d7e4454bec04193017278e5b42750c09e51e955ec5db5d70930851975e4d46"}
Sep 18 13:15:30.379038 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:30.378982    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cn6sm" podStartSLOduration=1.8520193900000002 podStartE2EDuration="4.378961713s" totalImagesPullingTime="2.526942323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:30.378210907 +0000 UTC m=+38.813087310" watchObservedRunningTime="2026-09-18 13:15:30.378961713 +0000 UTC m=+38.813838119"
Sep 18 13:15:33.049722 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.049649    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pg4dm"]
Sep 18 13:15:33.069838 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.069814    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pnzr7"]
Sep 18 13:15:33.069982 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.069944    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pg4dm"
Sep 18 13:15:33.073839 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.073817    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 18 13:15:33.073966 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.073895    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 13:15:33.075404 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.075382    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 13:15:33.075500 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.075423    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 18 13:15:33.075500 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.075456    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-68zdw\""
Sep 18 13:15:33.093763 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.093743    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-58qsl"]
Sep 18 13:15:33.093881 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.093863    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pnzr7"
Sep 18 13:15:33.110478 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.110454    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:15:33.110636 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.110485    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 18 13:15:33.110636 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.110575    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-vwh5m\""
Sep 18 13:15:33.113922 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.113903    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pg4dm"]
Sep 18 13:15:33.114037 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.114002    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-58qsl"
Sep 18 13:15:33.116464 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.116439    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 18 13:15:33.116629 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.116609    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-25zhs\""
Sep 18 13:15:33.117769 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.117747    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 18 13:15:33.117865 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.117816    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 18 13:15:33.118040 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.118018    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:15:33.125860 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.125841    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pnzr7"]
Sep 18 13:15:33.126646 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.126630    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-58qsl"]
Sep 18 13:15:33.153414 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.153298    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6rdg\" (UniqueName: \"kubernetes.io/projected/6b914660-2e06-4c5f-897f-f3ac5b12c348-kube-api-access-j6rdg\") pod \"cluster-monitoring-operator-847b8bd8d7-pg4dm\" (UID: \"6b914660-2e06-4c5f-897f-f3ac5b12c348\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pg4dm"
Sep 18 13:15:33.153414 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.153346    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b914660-2e06-4c5f-897f-f3ac5b12c348-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-pg4dm\" (UID: \"6b914660-2e06-4c5f-897f-f3ac5b12c348\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pg4dm"
Sep 18 13:15:33.153414 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.153397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6b914660-2e06-4c5f-897f-f3ac5b12c348-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-pg4dm\" (UID: \"6b914660-2e06-4c5f-897f-f3ac5b12c348\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pg4dm"
Sep 18 13:15:33.154631 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.154607    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-ggbng"]
Sep 18 13:15:33.180319 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.180293    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-c9tj5"]
Sep 18 13:15:33.180447 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.180434    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-ggbng"
Sep 18 13:15:33.186222 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.186200    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 18 13:15:33.186326 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.186237    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 18 13:15:33.186733 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.186713    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:15:33.186828 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.186803    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-wdfqf\""
Sep 18 13:15:33.200026 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.199998    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-ggbng"]
Sep 18 13:15:33.200145 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.200034    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-c9tj5"]
Sep 18 13:15:33.200145 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.200071    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-c9tj5"
Sep 18 13:15:33.203958 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.203943    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 18 13:15:33.204042 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.203991    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 18 13:15:33.204240 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.204226    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 18 13:15:33.205616 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.205581    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:15:33.205703 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.205679    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-nxwck\""
Sep 18 13:15:33.254380 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.254350    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b914660-2e06-4c5f-897f-f3ac5b12c348-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-pg4dm\" (UID: \"6b914660-2e06-4c5f-897f-f3ac5b12c348\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pg4dm"
Sep 18 13:15:33.254380 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.254385    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7cd87f4-e7a0-431e-9cd9-73fbe25e3a7f-serving-cert\") pod \"service-ca-operator-57cdc4b768-58qsl\" (UID: \"c7cd87f4-e7a0-431e-9cd9-73fbe25e3a7f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-58qsl"
Sep 18 13:15:33.254588 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.254411    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6b914660-2e06-4c5f-897f-f3ac5b12c348-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-pg4dm\" (UID: \"6b914660-2e06-4c5f-897f-f3ac5b12c348\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pg4dm"
Sep 18 13:15:33.254588 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.254438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mfdr\" (UniqueName: \"kubernetes.io/projected/cfad07b4-8ee8-4ce6-81bf-5bdc3f601aa0-kube-api-access-7mfdr\") pod \"volume-data-source-validator-5c4db8bdf9-pnzr7\" (UID: \"cfad07b4-8ee8-4ce6-81bf-5bdc3f601aa0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pnzr7"
Sep 18 13:15:33.254588 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.254472    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-ggbng\" (UID: \"b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-ggbng"
Sep 18 13:15:33.254588 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:33.254517    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 13:15:33.254588 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.254590    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km6w2\" (UniqueName: \"kubernetes.io/projected/b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0-kube-api-access-km6w2\") pod \"cluster-samples-operator-685d7d68b5-ggbng\" (UID: \"b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-ggbng"
Sep 18 13:15:33.254788 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.254645    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c7cd87f4-e7a0-431e-9cd9-73fbe25e3a7f-config\") pod \"service-ca-operator-57cdc4b768-58qsl\" (UID: \"c7cd87f4-e7a0-431e-9cd9-73fbe25e3a7f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-58qsl"
Sep 18 13:15:33.254788 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.254666    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6rdg\" (UniqueName: \"kubernetes.io/projected/6b914660-2e06-4c5f-897f-f3ac5b12c348-kube-api-access-j6rdg\") pod \"cluster-monitoring-operator-847b8bd8d7-pg4dm\" (UID: \"6b914660-2e06-4c5f-897f-f3ac5b12c348\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pg4dm"
Sep 18 13:15:33.254788 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.254688    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8lmh\" (UniqueName: \"kubernetes.io/projected/c7cd87f4-e7a0-431e-9cd9-73fbe25e3a7f-kube-api-access-m8lmh\") pod \"service-ca-operator-57cdc4b768-58qsl\" (UID: \"c7cd87f4-e7a0-431e-9cd9-73fbe25e3a7f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-58qsl"
Sep 18 13:15:33.254788 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:33.254743    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b914660-2e06-4c5f-897f-f3ac5b12c348-cluster-monitoring-operator-tls podName:6b914660-2e06-4c5f-897f-f3ac5b12c348 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:33.75472876 +0000 UTC m=+42.189605146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b914660-2e06-4c5f-897f-f3ac5b12c348-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-pg4dm" (UID: "6b914660-2e06-4c5f-897f-f3ac5b12c348") : secret "cluster-monitoring-operator-tls" not found
Sep 18 13:15:33.255121 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.255102    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6b914660-2e06-4c5f-897f-f3ac5b12c348-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-pg4dm\" (UID: \"6b914660-2e06-4c5f-897f-f3ac5b12c348\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pg4dm"
Sep 18 13:15:33.308615 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.308509    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-mvmdl"]
Sep 18 13:15:33.330057 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.330028    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-k86vn"]
Sep 18 13:15:33.330207 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.330158    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-mvmdl"
Sep 18 13:15:33.349667 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.349648    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-t7tk9\""
Sep 18 13:15:33.350346 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.350321    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-k86vn"
Sep 18 13:15:33.353315 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.353296    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6rdg\" (UniqueName: \"kubernetes.io/projected/6b914660-2e06-4c5f-897f-f3ac5b12c348-kube-api-access-j6rdg\") pod \"cluster-monitoring-operator-847b8bd8d7-pg4dm\" (UID: \"6b914660-2e06-4c5f-897f-f3ac5b12c348\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pg4dm"
Sep 18 13:15:33.353480 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.353448    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 18 13:15:33.353568 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.353555    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 18 13:15:33.354010 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.353990    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 18 13:15:33.354239 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.354220    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:15:33.354308 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.354238    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-pwwxh\""
Sep 18 13:15:33.355130 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.355111    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c7cd87f4-e7a0-431e-9cd9-73fbe25e3a7f-config\") pod \"service-ca-operator-57cdc4b768-58qsl\" (UID: \"c7cd87f4-e7a0-431e-9cd9-73fbe25e3a7f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-58qsl"
Sep 18 13:15:33.355190 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.355150    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1555f97d-9829-4708-b2c7-b2afb963ad05-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-c9tj5\" (UID: \"1555f97d-9829-4708-b2c7-b2afb963ad05\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-c9tj5"
Sep 18 13:15:33.355190 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.355184    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8lmh\" (UniqueName: \"kubernetes.io/projected/c7cd87f4-e7a0-431e-9cd9-73fbe25e3a7f-kube-api-access-m8lmh\") pod \"service-ca-operator-57cdc4b768-58qsl\" (UID: \"c7cd87f4-e7a0-431e-9cd9-73fbe25e3a7f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-58qsl"
Sep 18 13:15:33.355279 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.355210    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1555f97d-9829-4708-b2c7-b2afb963ad05-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-c9tj5\" (UID: \"1555f97d-9829-4708-b2c7-b2afb963ad05\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-c9tj5"
Sep 18 13:15:33.355279 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.355255    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7cd87f4-e7a0-431e-9cd9-73fbe25e3a7f-serving-cert\") pod \"service-ca-operator-57cdc4b768-58qsl\" (UID: \"c7cd87f4-e7a0-431e-9cd9-73fbe25e3a7f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-58qsl"
Sep 18 13:15:33.355395 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.355344    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mfdr\" (UniqueName: \"kubernetes.io/projected/cfad07b4-8ee8-4ce6-81bf-5bdc3f601aa0-kube-api-access-7mfdr\") pod \"volume-data-source-validator-5c4db8bdf9-pnzr7\" (UID: \"cfad07b4-8ee8-4ce6-81bf-5bdc3f601aa0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pnzr7"
Sep 18 13:15:33.355395 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.355376    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-ggbng\" (UID: \"b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-ggbng"
Sep 18 13:15:33.355534 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.355413    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7zhk\" (UniqueName: \"kubernetes.io/projected/1555f97d-9829-4708-b2c7-b2afb963ad05-kube-api-access-m7zhk\") pod \"kube-storage-version-migrator-operator-55889f7b47-c9tj5\" (UID: \"1555f97d-9829-4708-b2c7-b2afb963ad05\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-c9tj5"
Sep 18 13:15:33.355534 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:33.355486    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 13:15:33.355534 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.355490    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km6w2\" (UniqueName: \"kubernetes.io/projected/b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0-kube-api-access-km6w2\") pod \"cluster-samples-operator-685d7d68b5-ggbng\" (UID: \"b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-ggbng"
Sep 18 13:15:33.355534 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:33.355551    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0-samples-operator-tls podName:b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:33.855534169 +0000 UTC m=+42.290410575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-ggbng" (UID: "b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0") : secret "samples-operator-tls" not found
Sep 18 13:15:33.355923 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.355714    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c7cd87f4-e7a0-431e-9cd9-73fbe25e3a7f-config\") pod \"service-ca-operator-57cdc4b768-58qsl\" (UID: \"c7cd87f4-e7a0-431e-9cd9-73fbe25e3a7f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-58qsl"
Sep 18 13:15:33.357974 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.357950    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7cd87f4-e7a0-431e-9cd9-73fbe25e3a7f-serving-cert\") pod \"service-ca-operator-57cdc4b768-58qsl\" (UID: \"c7cd87f4-e7a0-431e-9cd9-73fbe25e3a7f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-58qsl"
Sep 18 13:15:33.363276 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.363255    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 18 13:15:33.369230 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.369211    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-k86vn"]
Sep 18 13:15:33.383472 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.383431    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-mvmdl"]
Sep 18 13:15:33.411751 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.411726    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8lmh\" (UniqueName: \"kubernetes.io/projected/c7cd87f4-e7a0-431e-9cd9-73fbe25e3a7f-kube-api-access-m8lmh\") pod \"service-ca-operator-57cdc4b768-58qsl\" (UID: \"c7cd87f4-e7a0-431e-9cd9-73fbe25e3a7f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-58qsl"
Sep 18 13:15:33.414055 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.414029    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-668879999c-dfvz7"]
Sep 18 13:15:33.415776 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.415759    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mfdr\" (UniqueName: \"kubernetes.io/projected/cfad07b4-8ee8-4ce6-81bf-5bdc3f601aa0-kube-api-access-7mfdr\") pod \"volume-data-source-validator-5c4db8bdf9-pnzr7\" (UID: \"cfad07b4-8ee8-4ce6-81bf-5bdc3f601aa0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pnzr7"
Sep 18 13:15:33.422454 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.422434    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-58qsl"
Sep 18 13:15:33.428606 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.428576    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km6w2\" (UniqueName: \"kubernetes.io/projected/b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0-kube-api-access-km6w2\") pod \"cluster-samples-operator-685d7d68b5-ggbng\" (UID: \"b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-ggbng"
Sep 18 13:15:33.438361 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.438343    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:33.441197 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.441182    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 13:15:33.441404 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.441389    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 13:15:33.441623 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.441605    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 13:15:33.441685 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.441650    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w5nbx\""
Sep 18 13:15:33.452002 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.451980    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-668879999c-dfvz7"]
Sep 18 13:15:33.456105 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.456083    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/55e33e14-3f7a-4793-a724-9fd5022740d7-serving-cert\") pod \"console-operator-fd4bb46c-k86vn\" (UID: \"55e33e14-3f7a-4793-a724-9fd5022740d7\") " pod="openshift-console-operator/console-operator-fd4bb46c-k86vn"
Sep 18 13:15:33.456205 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.456119    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1555f97d-9829-4708-b2c7-b2afb963ad05-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-c9tj5\" (UID: \"1555f97d-9829-4708-b2c7-b2afb963ad05\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-c9tj5"
Sep 18 13:15:33.456205 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.456141    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1555f97d-9829-4708-b2c7-b2afb963ad05-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-c9tj5\" (UID: \"1555f97d-9829-4708-b2c7-b2afb963ad05\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-c9tj5"
Sep 18 13:15:33.456297 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.456219    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/55e33e14-3f7a-4793-a724-9fd5022740d7-config\") pod \"console-operator-fd4bb46c-k86vn\" (UID: \"55e33e14-3f7a-4793-a724-9fd5022740d7\") " pod="openshift-console-operator/console-operator-fd4bb46c-k86vn"
Sep 18 13:15:33.456297 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.456251    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjzxc\" (UniqueName: \"kubernetes.io/projected/55e33e14-3f7a-4793-a724-9fd5022740d7-kube-api-access-xjzxc\") pod \"console-operator-fd4bb46c-k86vn\" (UID: \"55e33e14-3f7a-4793-a724-9fd5022740d7\") " pod="openshift-console-operator/console-operator-fd4bb46c-k86vn"
Sep 18 13:15:33.456402 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.456309    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7zhk\" (UniqueName: \"kubernetes.io/projected/1555f97d-9829-4708-b2c7-b2afb963ad05-kube-api-access-m7zhk\") pod \"kube-storage-version-migrator-operator-55889f7b47-c9tj5\" (UID: \"1555f97d-9829-4708-b2c7-b2afb963ad05\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-c9tj5"
Sep 18 13:15:33.456402 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.456337    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55e33e14-3f7a-4793-a724-9fd5022740d7-trusted-ca\") pod \"console-operator-fd4bb46c-k86vn\" (UID: \"55e33e14-3f7a-4793-a724-9fd5022740d7\") " pod="openshift-console-operator/console-operator-fd4bb46c-k86vn"
Sep 18 13:15:33.456499 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.456467    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tvbr\" (UniqueName: \"kubernetes.io/projected/bd1f5ab6-1c66-485b-9135-45b726ae935e-kube-api-access-8tvbr\") pod \"network-check-source-7cbc45fddc-mvmdl\" (UID: \"bd1f5ab6-1c66-485b-9135-45b726ae935e\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-mvmdl"
Sep 18 13:15:33.456680 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.456661    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1555f97d-9829-4708-b2c7-b2afb963ad05-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-c9tj5\" (UID: \"1555f97d-9829-4708-b2c7-b2afb963ad05\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-c9tj5"
Sep 18 13:15:33.458313 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.458295    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1555f97d-9829-4708-b2c7-b2afb963ad05-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-c9tj5\" (UID: \"1555f97d-9829-4708-b2c7-b2afb963ad05\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-c9tj5"
Sep 18 13:15:33.460207 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.460190    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 13:15:33.471557 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.471524    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7zhk\" (UniqueName: \"kubernetes.io/projected/1555f97d-9829-4708-b2c7-b2afb963ad05-kube-api-access-m7zhk\") pod \"kube-storage-version-migrator-operator-55889f7b47-c9tj5\" (UID: \"1555f97d-9829-4708-b2c7-b2afb963ad05\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-c9tj5"
Sep 18 13:15:33.508049 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.508021    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-c9tj5"
Sep 18 13:15:33.557139 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.557116    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0bb07b38-9199-424d-9fb7-3ffa57def30a-installation-pull-secrets\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:33.557263 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.557148    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/55e33e14-3f7a-4793-a724-9fd5022740d7-config\") pod \"console-operator-fd4bb46c-k86vn\" (UID: \"55e33e14-3f7a-4793-a724-9fd5022740d7\") " pod="openshift-console-operator/console-operator-fd4bb46c-k86vn"
Sep 18 13:15:33.557263 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.557174    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0bb07b38-9199-424d-9fb7-3ffa57def30a-image-registry-private-configuration\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:33.557263 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.557218    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0bb07b38-9199-424d-9fb7-3ffa57def30a-ca-trust-extracted\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:33.557263 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.557255    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tvbr\" (UniqueName: \"kubernetes.io/projected/bd1f5ab6-1c66-485b-9135-45b726ae935e-kube-api-access-8tvbr\") pod \"network-check-source-7cbc45fddc-mvmdl\" (UID: \"bd1f5ab6-1c66-485b-9135-45b726ae935e\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-mvmdl"
Sep 18 13:15:33.557503 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.557290    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnmrq\" (UniqueName: \"kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-kube-api-access-fnmrq\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:33.557503 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.557315    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0bb07b38-9199-424d-9fb7-3ffa57def30a-trusted-ca\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:33.557503 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.557360    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjzxc\" (UniqueName: \"kubernetes.io/projected/55e33e14-3f7a-4793-a724-9fd5022740d7-kube-api-access-xjzxc\") pod \"console-operator-fd4bb46c-k86vn\" (UID: \"55e33e14-3f7a-4793-a724-9fd5022740d7\") " pod="openshift-console-operator/console-operator-fd4bb46c-k86vn"
Sep 18 13:15:33.557503 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.557454    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-tls\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:33.557503 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.557500    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/55e33e14-3f7a-4793-a724-9fd5022740d7-serving-cert\") pod \"console-operator-fd4bb46c-k86vn\" (UID: \"55e33e14-3f7a-4793-a724-9fd5022740d7\") " pod="openshift-console-operator/console-operator-fd4bb46c-k86vn"
Sep 18 13:15:33.557821 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.557535    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55e33e14-3f7a-4793-a724-9fd5022740d7-trusted-ca\") pod \"console-operator-fd4bb46c-k86vn\" (UID: \"55e33e14-3f7a-4793-a724-9fd5022740d7\") " pod="openshift-console-operator/console-operator-fd4bb46c-k86vn"
Sep 18 13:15:33.557821 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.557560    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-certificates\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:33.557821 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.557620    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-bound-sa-token\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:33.557979 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.557953    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/55e33e14-3f7a-4793-a724-9fd5022740d7-config\") pod \"console-operator-fd4bb46c-k86vn\" (UID: \"55e33e14-3f7a-4793-a724-9fd5022740d7\") " pod="openshift-console-operator/console-operator-fd4bb46c-k86vn"
Sep 18 13:15:33.558222 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.558201    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55e33e14-3f7a-4793-a724-9fd5022740d7-trusted-ca\") pod \"console-operator-fd4bb46c-k86vn\" (UID: \"55e33e14-3f7a-4793-a724-9fd5022740d7\") " pod="openshift-console-operator/console-operator-fd4bb46c-k86vn"
Sep 18 13:15:33.559668 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.559624    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/55e33e14-3f7a-4793-a724-9fd5022740d7-serving-cert\") pod \"console-operator-fd4bb46c-k86vn\" (UID: \"55e33e14-3f7a-4793-a724-9fd5022740d7\") " pod="openshift-console-operator/console-operator-fd4bb46c-k86vn"
Sep 18 13:15:33.569444 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.569419    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tvbr\" (UniqueName: \"kubernetes.io/projected/bd1f5ab6-1c66-485b-9135-45b726ae935e-kube-api-access-8tvbr\") pod \"network-check-source-7cbc45fddc-mvmdl\" (UID: \"bd1f5ab6-1c66-485b-9135-45b726ae935e\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-mvmdl"
Sep 18 13:15:33.571261 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.571241    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjzxc\" (UniqueName: \"kubernetes.io/projected/55e33e14-3f7a-4793-a724-9fd5022740d7-kube-api-access-xjzxc\") pod \"console-operator-fd4bb46c-k86vn\" (UID: \"55e33e14-3f7a-4793-a724-9fd5022740d7\") " pod="openshift-console-operator/console-operator-fd4bb46c-k86vn"
Sep 18 13:15:33.638137 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.638100    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-mvmdl"
Sep 18 13:15:33.658455 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.658429    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-certificates\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:33.658562 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.658479    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-bound-sa-token\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:33.658562 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.658500    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0bb07b38-9199-424d-9fb7-3ffa57def30a-installation-pull-secrets\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:33.658562 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.658528    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0bb07b38-9199-424d-9fb7-3ffa57def30a-image-registry-private-configuration\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:33.658562 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.658547    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0bb07b38-9199-424d-9fb7-3ffa57def30a-ca-trust-extracted\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:33.658765 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.658587    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnmrq\" (UniqueName: \"kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-kube-api-access-fnmrq\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:33.658765 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.658635    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0bb07b38-9199-424d-9fb7-3ffa57def30a-trusted-ca\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:33.658765 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.658691    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-tls\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:33.658905 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:33.658783    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:15:33.658905 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:33.658798    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668879999c-dfvz7: secret "image-registry-tls" not found
Sep 18 13:15:33.658905 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:33.658847    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-tls podName:0bb07b38-9199-424d-9fb7-3ffa57def30a nodeName:}" failed. No retries permitted until 2026-09-18 13:15:34.158829752 +0000 UTC m=+42.593706154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-tls") pod "image-registry-668879999c-dfvz7" (UID: "0bb07b38-9199-424d-9fb7-3ffa57def30a") : secret "image-registry-tls" not found
Sep 18 13:15:33.659072 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.658999    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0bb07b38-9199-424d-9fb7-3ffa57def30a-ca-trust-extracted\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:33.659181 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.659151    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-certificates\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:33.659799 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.659762    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0bb07b38-9199-424d-9fb7-3ffa57def30a-trusted-ca\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:33.660887 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.660867    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0bb07b38-9199-424d-9fb7-3ffa57def30a-installation-pull-secrets\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:33.660966 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.660915    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0bb07b38-9199-424d-9fb7-3ffa57def30a-image-registry-private-configuration\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:33.669947 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.669922    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-k86vn"
Sep 18 13:15:33.673283 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.673173    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-bound-sa-token\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:33.673283 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.673268    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnmrq\" (UniqueName: \"kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-kube-api-access-fnmrq\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:33.701700 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.701674    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pnzr7"
Sep 18 13:15:33.725637 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.725584    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-58qsl"]
Sep 18 13:15:33.728471 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:15:33.728425    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7cd87f4_e7a0_431e_9cd9_73fbe25e3a7f.slice/crio-1c51da0d515047f10f4471ba9a4918c58c98932d719885655f5721a7f1af3ae6 WatchSource:0}: Error finding container 1c51da0d515047f10f4471ba9a4918c58c98932d719885655f5721a7f1af3ae6: Status 404 returned error can't find the container with id 1c51da0d515047f10f4471ba9a4918c58c98932d719885655f5721a7f1af3ae6
Sep 18 13:15:33.759296 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.759267    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b914660-2e06-4c5f-897f-f3ac5b12c348-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-pg4dm\" (UID: \"6b914660-2e06-4c5f-897f-f3ac5b12c348\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pg4dm"
Sep 18 13:15:33.759447 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:33.759387    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 13:15:33.759507 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:33.759451    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b914660-2e06-4c5f-897f-f3ac5b12c348-cluster-monitoring-operator-tls podName:6b914660-2e06-4c5f-897f-f3ac5b12c348 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:34.759436411 +0000 UTC m=+43.194312811 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b914660-2e06-4c5f-897f-f3ac5b12c348-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-pg4dm" (UID: "6b914660-2e06-4c5f-897f-f3ac5b12c348") : secret "cluster-monitoring-operator-tls" not found
Sep 18 13:15:33.824522 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.824430    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-c9tj5"]
Sep 18 13:15:33.828300 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:15:33.828275    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1555f97d_9829_4708_b2c7_b2afb963ad05.slice/crio-92c7cba3dfb0cf476f2eac341a4262cc02a7656a23fc38a00eb06a441bcb06c1 WatchSource:0}: Error finding container 92c7cba3dfb0cf476f2eac341a4262cc02a7656a23fc38a00eb06a441bcb06c1: Status 404 returned error can't find the container with id 92c7cba3dfb0cf476f2eac341a4262cc02a7656a23fc38a00eb06a441bcb06c1
Sep 18 13:15:33.860315 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.860290    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-ggbng\" (UID: \"b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-ggbng"
Sep 18 13:15:33.860462 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:33.860439    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 13:15:33.860513 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:33.860499    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0-samples-operator-tls podName:b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:34.860484889 +0000 UTC m=+43.295361272 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-ggbng" (UID: "b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0") : secret "samples-operator-tls" not found
Sep 18 13:15:33.948082 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.948050    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-mvmdl"]
Sep 18 13:15:33.951392 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:15:33.951368    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1f5ab6_1c66_485b_9135_45b726ae935e.slice/crio-128b0967cb539f3d7536c0c90ee2d90152e66ebc3329dd9bfb345bbb428f2a1a WatchSource:0}: Error finding container 128b0967cb539f3d7536c0c90ee2d90152e66ebc3329dd9bfb345bbb428f2a1a: Status 404 returned error can't find the container with id 128b0967cb539f3d7536c0c90ee2d90152e66ebc3329dd9bfb345bbb428f2a1a
Sep 18 13:15:33.978821 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:33.978795    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-k86vn"]
Sep 18 13:15:33.982119 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:15:33.982097    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e33e14_3f7a_4793_a724_9fd5022740d7.slice/crio-34fea53979a28103d32e5d086c39ef8070adb5d6005eba98233c874833c2d82f WatchSource:0}: Error finding container 34fea53979a28103d32e5d086c39ef8070adb5d6005eba98233c874833c2d82f: Status 404 returned error can't find the container with id 34fea53979a28103d32e5d086c39ef8070adb5d6005eba98233c874833c2d82f
Sep 18 13:15:34.023138 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:34.023105    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pnzr7"]
Sep 18 13:15:34.028453 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:15:34.028431    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfad07b4_8ee8_4ce6_81bf_5bdc3f601aa0.slice/crio-85b91c3a6ac81da8e00264f9ef4255d0894ac1c53154d5500b0be1311a623525 WatchSource:0}: Error finding container 85b91c3a6ac81da8e00264f9ef4255d0894ac1c53154d5500b0be1311a623525: Status 404 returned error can't find the container with id 85b91c3a6ac81da8e00264f9ef4255d0894ac1c53154d5500b0be1311a623525
Sep 18 13:15:34.162255 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:34.162217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-tls\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:34.162726 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:34.162400    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:15:34.162726 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:34.162425    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668879999c-dfvz7: secret "image-registry-tls" not found
Sep 18 13:15:34.162726 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:34.162495    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-tls podName:0bb07b38-9199-424d-9fb7-3ffa57def30a nodeName:}" failed. No retries permitted until 2026-09-18 13:15:35.162475181 +0000 UTC m=+43.597351565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-tls") pod "image-registry-668879999c-dfvz7" (UID: "0bb07b38-9199-424d-9fb7-3ffa57def30a") : secret "image-registry-tls" not found
Sep 18 13:15:34.263560 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:34.263523    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96cd73fd-fcb1-44ea-b184-63a97a0277f4-metrics-tls\") pod \"dns-default-r94bv\" (UID: \"96cd73fd-fcb1-44ea-b184-63a97a0277f4\") " pod="openshift-dns/dns-default-r94bv"
Sep 18 13:15:34.263713 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:34.263687    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:15:34.263776 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:34.263755    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96cd73fd-fcb1-44ea-b184-63a97a0277f4-metrics-tls podName:96cd73fd-fcb1-44ea-b184-63a97a0277f4 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:42.263734353 +0000 UTC m=+50.698610740 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96cd73fd-fcb1-44ea-b184-63a97a0277f4-metrics-tls") pod "dns-default-r94bv" (UID: "96cd73fd-fcb1-44ea-b184-63a97a0277f4") : secret "dns-default-metrics-tls" not found
Sep 18 13:15:34.355948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:34.355911    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-k86vn" event={"ID":"55e33e14-3f7a-4793-a724-9fd5022740d7","Type":"ContainerStarted","Data":"34fea53979a28103d32e5d086c39ef8070adb5d6005eba98233c874833c2d82f"}
Sep 18 13:15:34.357163 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:34.357137    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pnzr7" event={"ID":"cfad07b4-8ee8-4ce6-81bf-5bdc3f601aa0","Type":"ContainerStarted","Data":"85b91c3a6ac81da8e00264f9ef4255d0894ac1c53154d5500b0be1311a623525"}
Sep 18 13:15:34.358352 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:34.358328    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-mvmdl" event={"ID":"bd1f5ab6-1c66-485b-9135-45b726ae935e","Type":"ContainerStarted","Data":"128b0967cb539f3d7536c0c90ee2d90152e66ebc3329dd9bfb345bbb428f2a1a"}
Sep 18 13:15:34.359509 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:34.359490    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-c9tj5" event={"ID":"1555f97d-9829-4708-b2c7-b2afb963ad05","Type":"ContainerStarted","Data":"92c7cba3dfb0cf476f2eac341a4262cc02a7656a23fc38a00eb06a441bcb06c1"}
Sep 18 13:15:34.360626 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:34.360606    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-58qsl" event={"ID":"c7cd87f4-e7a0-431e-9cd9-73fbe25e3a7f","Type":"ContainerStarted","Data":"1c51da0d515047f10f4471ba9a4918c58c98932d719885655f5721a7f1af3ae6"}
Sep 18 13:15:34.767043 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:34.766928    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b914660-2e06-4c5f-897f-f3ac5b12c348-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-pg4dm\" (UID: \"6b914660-2e06-4c5f-897f-f3ac5b12c348\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pg4dm"
Sep 18 13:15:34.767240 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:34.767193    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 13:15:34.767331 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:34.767263    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b914660-2e06-4c5f-897f-f3ac5b12c348-cluster-monitoring-operator-tls podName:6b914660-2e06-4c5f-897f-f3ac5b12c348 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:36.767245163 +0000 UTC m=+45.202121567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b914660-2e06-4c5f-897f-f3ac5b12c348-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-pg4dm" (UID: "6b914660-2e06-4c5f-897f-f3ac5b12c348") : secret "cluster-monitoring-operator-tls" not found
Sep 18 13:15:34.868366 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:34.868327    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-ggbng\" (UID: \"b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-ggbng"
Sep 18 13:15:34.868549 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:34.868499    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 13:15:34.868633 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:34.868561    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0-samples-operator-tls podName:b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:36.868542902 +0000 UTC m=+45.303419300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-ggbng" (UID: "b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0") : secret "samples-operator-tls" not found
Sep 18 13:15:35.171924 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:35.171341    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-tls\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:35.171924 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:35.171498    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:15:35.171924 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:35.171513    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668879999c-dfvz7: secret "image-registry-tls" not found
Sep 18 13:15:35.171924 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:35.171569    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-tls podName:0bb07b38-9199-424d-9fb7-3ffa57def30a nodeName:}" failed. No retries permitted until 2026-09-18 13:15:37.171551604 +0000 UTC m=+45.606427993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-tls") pod "image-registry-668879999c-dfvz7" (UID: "0bb07b38-9199-424d-9fb7-3ffa57def30a") : secret "image-registry-tls" not found
Sep 18 13:15:36.786100 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:36.786053    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b914660-2e06-4c5f-897f-f3ac5b12c348-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-pg4dm\" (UID: \"6b914660-2e06-4c5f-897f-f3ac5b12c348\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pg4dm"
Sep 18 13:15:36.786547 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:36.786234    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 13:15:36.786547 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:36.786315    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b914660-2e06-4c5f-897f-f3ac5b12c348-cluster-monitoring-operator-tls podName:6b914660-2e06-4c5f-897f-f3ac5b12c348 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:40.786293993 +0000 UTC m=+49.221170375 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b914660-2e06-4c5f-897f-f3ac5b12c348-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-pg4dm" (UID: "6b914660-2e06-4c5f-897f-f3ac5b12c348") : secret "cluster-monitoring-operator-tls" not found
Sep 18 13:15:36.887269 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:36.887228    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-ggbng\" (UID: \"b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-ggbng"
Sep 18 13:15:36.887460 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:36.887431    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 13:15:36.887538 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:36.887526    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0-samples-operator-tls podName:b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:40.887495049 +0000 UTC m=+49.322371452 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-ggbng" (UID: "b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0") : secret "samples-operator-tls" not found
Sep 18 13:15:37.189961 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:37.189919    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-tls\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:37.190153 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:37.190092    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:15:37.190153 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:37.190115    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668879999c-dfvz7: secret "image-registry-tls" not found
Sep 18 13:15:37.190262 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:37.190182    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-tls podName:0bb07b38-9199-424d-9fb7-3ffa57def30a nodeName:}" failed. No retries permitted until 2026-09-18 13:15:41.190162682 +0000 UTC m=+49.625039084 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-tls") pod "image-registry-668879999c-dfvz7" (UID: "0bb07b38-9199-424d-9fb7-3ffa57def30a") : secret "image-registry-tls" not found
Sep 18 13:15:37.215798 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:37.215766    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xjrc4"]
Sep 18 13:15:37.240611 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:37.240564    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:37.244001 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:37.243976    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 13:15:37.244869 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:37.244852    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 13:15:37.244869 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:37.244867    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 13:15:37.245180 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:37.245163    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 13:15:37.250647 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:37.250627    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p2lqc\""
Sep 18 13:15:37.256249 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:37.256228    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xjrc4"]
Sep 18 13:15:37.392318 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:37.392275    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3d630003-2bfd-4cb9-9b8a-745c0ccca548-crio-socket\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:37.392499 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:37.392468    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47vgw\" (UniqueName: \"kubernetes.io/projected/3d630003-2bfd-4cb9-9b8a-745c0ccca548-kube-api-access-47vgw\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:37.392563 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:37.392517    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d630003-2bfd-4cb9-9b8a-745c0ccca548-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:37.392645 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:37.392620    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d630003-2bfd-4cb9-9b8a-745c0ccca548-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:37.392715 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:37.392652    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d630003-2bfd-4cb9-9b8a-745c0ccca548-data-volume\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:37.494021 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:37.493941    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d630003-2bfd-4cb9-9b8a-745c0ccca548-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:37.494021 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:37.493997    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d630003-2bfd-4cb9-9b8a-745c0ccca548-data-volume\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:37.494265 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:37.494030    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3d630003-2bfd-4cb9-9b8a-745c0ccca548-crio-socket\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:37.494265 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:37.494097    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:15:37.494265 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:37.494155    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vgw\" (UniqueName: \"kubernetes.io/projected/3d630003-2bfd-4cb9-9b8a-745c0ccca548-kube-api-access-47vgw\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:37.494265 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:37.494190    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d630003-2bfd-4cb9-9b8a-745c0ccca548-kube-rbac-proxy-cm podName:3d630003-2bfd-4cb9-9b8a-745c0ccca548 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:37.99416801 +0000 UTC m=+46.429044395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3d630003-2bfd-4cb9-9b8a-745c0ccca548-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xjrc4" (UID: "3d630003-2bfd-4cb9-9b8a-745c0ccca548") : configmap "kube-rbac-proxy" not found
Sep 18 13:15:37.494265 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:37.494237    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d630003-2bfd-4cb9-9b8a-745c0ccca548-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:37.494512 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:37.494312    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3d630003-2bfd-4cb9-9b8a-745c0ccca548-crio-socket\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:37.494512 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:37.494381    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:15:37.494512 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:37.494394    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d630003-2bfd-4cb9-9b8a-745c0ccca548-data-volume\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:37.494512 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:37.494469    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d630003-2bfd-4cb9-9b8a-745c0ccca548-insights-runtime-extractor-tls podName:3d630003-2bfd-4cb9-9b8a-745c0ccca548 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:37.994448984 +0000 UTC m=+46.429325388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d630003-2bfd-4cb9-9b8a-745c0ccca548-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xjrc4" (UID: "3d630003-2bfd-4cb9-9b8a-745c0ccca548") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:15:37.513261 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:37.513229    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47vgw\" (UniqueName: \"kubernetes.io/projected/3d630003-2bfd-4cb9-9b8a-745c0ccca548-kube-api-access-47vgw\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:37.998862 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:37.998820    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d630003-2bfd-4cb9-9b8a-745c0ccca548-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:37.999267 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:37.998935    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d630003-2bfd-4cb9-9b8a-745c0ccca548-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:37.999267 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:37.998983    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:15:37.999267 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:37.999053    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:15:37.999267 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:37.999080    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d630003-2bfd-4cb9-9b8a-745c0ccca548-kube-rbac-proxy-cm podName:3d630003-2bfd-4cb9-9b8a-745c0ccca548 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:38.999059522 +0000 UTC m=+47.433935906 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3d630003-2bfd-4cb9-9b8a-745c0ccca548-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xjrc4" (UID: "3d630003-2bfd-4cb9-9b8a-745c0ccca548") : configmap "kube-rbac-proxy" not found
Sep 18 13:15:37.999267 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:37.999124    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d630003-2bfd-4cb9-9b8a-745c0ccca548-insights-runtime-extractor-tls podName:3d630003-2bfd-4cb9-9b8a-745c0ccca548 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:38.999109107 +0000 UTC m=+47.433985489 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d630003-2bfd-4cb9-9b8a-745c0ccca548-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xjrc4" (UID: "3d630003-2bfd-4cb9-9b8a-745c0ccca548") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:15:38.911504 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:38.911475    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rd9fl_c6e79ce7-2e57-4103-871f-5b11f0a468d5/dns-node-resolver/0.log"
Sep 18 13:15:39.008135 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:39.008101    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d630003-2bfd-4cb9-9b8a-745c0ccca548-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:39.008502 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:39.008179    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d630003-2bfd-4cb9-9b8a-745c0ccca548-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:39.008502 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:39.008256    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:15:39.008502 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:39.008269    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:15:39.008502 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:39.008318    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d630003-2bfd-4cb9-9b8a-745c0ccca548-kube-rbac-proxy-cm podName:3d630003-2bfd-4cb9-9b8a-745c0ccca548 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:41.008300013 +0000 UTC m=+49.443176415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3d630003-2bfd-4cb9-9b8a-745c0ccca548-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xjrc4" (UID: "3d630003-2bfd-4cb9-9b8a-745c0ccca548") : configmap "kube-rbac-proxy" not found
Sep 18 13:15:39.008502 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:39.008358    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d630003-2bfd-4cb9-9b8a-745c0ccca548-insights-runtime-extractor-tls podName:3d630003-2bfd-4cb9-9b8a-745c0ccca548 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:41.008332004 +0000 UTC m=+49.443208402 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d630003-2bfd-4cb9-9b8a-745c0ccca548-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xjrc4" (UID: "3d630003-2bfd-4cb9-9b8a-745c0ccca548") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:15:39.532040 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:39.532017    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lglgb_df2366e7-fd3e-49d3-94af-686a9a9f262f/node-ca/0.log"
Sep 18 13:15:40.312267 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:40.312233    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cn6sm_2452828a-bce2-4da3-a477-88d9d048a4b3/serve-healthcheck-canary/0.log"
Sep 18 13:15:40.378449 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:40.378409    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-mvmdl" event={"ID":"bd1f5ab6-1c66-485b-9135-45b726ae935e","Type":"ContainerStarted","Data":"3dd18c522be33fe7c32e83a170d22e34d7ad426131e6489b5d97c89364440a29"}
Sep 18 13:15:40.380103 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:40.380075    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-c9tj5" event={"ID":"1555f97d-9829-4708-b2c7-b2afb963ad05","Type":"ContainerStarted","Data":"cb5e0ffc181b64762c7c732b900b0743c8aa6b6e8e67e5dff33590ad4b861dce"}
Sep 18 13:15:40.381622 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:40.381578    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-58qsl" event={"ID":"c7cd87f4-e7a0-431e-9cd9-73fbe25e3a7f","Type":"ContainerStarted","Data":"3005a5463449ca9ef6b20740e29be5f9fdd0366131eaa23623fc440eb9f82ad7"}
Sep 18 13:15:40.383275 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:40.383258    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k86vn_55e33e14-3f7a-4793-a724-9fd5022740d7/console-operator/0.log"
Sep 18 13:15:40.383354 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:40.383310    2566 generic.go:356] "Generic (PLEG): container finished" podID="55e33e14-3f7a-4793-a724-9fd5022740d7" containerID="36e9e0ebae15cc1952311bd883a581bb9a701df2a5cc1b732f7fc4b07f1dc50a" exitCode=255
Sep 18 13:15:40.383399 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:40.383370    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-k86vn" event={"ID":"55e33e14-3f7a-4793-a724-9fd5022740d7","Type":"ContainerDied","Data":"36e9e0ebae15cc1952311bd883a581bb9a701df2a5cc1b732f7fc4b07f1dc50a"}
Sep 18 13:15:40.383545 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:40.383530    2566 scope.go:117] "RemoveContainer" containerID="36e9e0ebae15cc1952311bd883a581bb9a701df2a5cc1b732f7fc4b07f1dc50a"
Sep 18 13:15:40.384917 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:40.384895    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pnzr7" event={"ID":"cfad07b4-8ee8-4ce6-81bf-5bdc3f601aa0","Type":"ContainerStarted","Data":"83961e2e647937bcba83784c3700f781b8239c7a98fcb7960672e0cb3b831bf3"}
Sep 18 13:15:40.399946 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:40.399900    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-mvmdl" podStartSLOduration=1.960663982 podStartE2EDuration="7.399883663s" totalImagesPullingTime="5.439219681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:40.399354876 +0000 UTC m=+48.834231279" watchObservedRunningTime="2026-09-18 13:15:40.399883663 +0000 UTC m=+48.834760068"
Sep 18 13:15:40.429527 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:40.429482    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-58qsl" podStartSLOduration=1.779385977 podStartE2EDuration="7.429466901s" totalImagesPullingTime="5.650080924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:40.428669738 +0000 UTC m=+48.863546142" watchObservedRunningTime="2026-09-18 13:15:40.429466901 +0000 UTC m=+48.864343305"
Sep 18 13:15:40.464582 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:40.464526    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-c9tj5" podStartSLOduration=1.911317291 podStartE2EDuration="7.464505378s" totalImagesPullingTime="5.553188087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:40.463675136 +0000 UTC m=+48.898551532" watchObservedRunningTime="2026-09-18 13:15:40.464505378 +0000 UTC m=+48.899381783"
Sep 18 13:15:40.531494 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:40.531437    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-pnzr7" podStartSLOduration=2.179800081 podStartE2EDuration="7.531419071s" totalImagesPullingTime="5.35161899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:40.530765416 +0000 UTC m=+48.965641832" watchObservedRunningTime="2026-09-18 13:15:40.531419071 +0000 UTC m=+48.966295476"
Sep 18 13:15:40.825918 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:40.825817    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b914660-2e06-4c5f-897f-f3ac5b12c348-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-pg4dm\" (UID: \"6b914660-2e06-4c5f-897f-f3ac5b12c348\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pg4dm"
Sep 18 13:15:40.826081 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:40.825964    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 13:15:40.826081 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:40.826032    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b914660-2e06-4c5f-897f-f3ac5b12c348-cluster-monitoring-operator-tls podName:6b914660-2e06-4c5f-897f-f3ac5b12c348 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:48.826015416 +0000 UTC m=+57.260891803 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b914660-2e06-4c5f-897f-f3ac5b12c348-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-pg4dm" (UID: "6b914660-2e06-4c5f-897f-f3ac5b12c348") : secret "cluster-monitoring-operator-tls" not found
Sep 18 13:15:40.926965 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:40.926926    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-ggbng\" (UID: \"b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-ggbng"
Sep 18 13:15:40.927124 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:40.927075    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 13:15:40.927164 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:40.927138    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0-samples-operator-tls podName:b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:48.927122642 +0000 UTC m=+57.361999024 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-ggbng" (UID: "b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0") : secret "samples-operator-tls" not found
Sep 18 13:15:41.028227 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:41.028183    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d630003-2bfd-4cb9-9b8a-745c0ccca548-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:41.028383 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:41.028260    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d630003-2bfd-4cb9-9b8a-745c0ccca548-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:41.028383 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:41.028350    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:15:41.028459 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:41.028398    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:15:41.028459 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:41.028412    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d630003-2bfd-4cb9-9b8a-745c0ccca548-insights-runtime-extractor-tls podName:3d630003-2bfd-4cb9-9b8a-745c0ccca548 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:45.02839747 +0000 UTC m=+53.463273860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d630003-2bfd-4cb9-9b8a-745c0ccca548-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xjrc4" (UID: "3d630003-2bfd-4cb9-9b8a-745c0ccca548") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:15:41.028459 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:41.028451    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d630003-2bfd-4cb9-9b8a-745c0ccca548-kube-rbac-proxy-cm podName:3d630003-2bfd-4cb9-9b8a-745c0ccca548 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:45.028438383 +0000 UTC m=+53.463314765 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3d630003-2bfd-4cb9-9b8a-745c0ccca548-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xjrc4" (UID: "3d630003-2bfd-4cb9-9b8a-745c0ccca548") : configmap "kube-rbac-proxy" not found
Sep 18 13:15:41.229808 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:41.229769    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-tls\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:41.229993 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:41.229894    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:15:41.229993 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:41.229906    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668879999c-dfvz7: secret "image-registry-tls" not found
Sep 18 13:15:41.229993 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:41.229960    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-tls podName:0bb07b38-9199-424d-9fb7-3ffa57def30a nodeName:}" failed. No retries permitted until 2026-09-18 13:15:49.229947102 +0000 UTC m=+57.664823484 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-tls") pod "image-registry-668879999c-dfvz7" (UID: "0bb07b38-9199-424d-9fb7-3ffa57def30a") : secret "image-registry-tls" not found
Sep 18 13:15:41.389875 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:41.389850    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k86vn_55e33e14-3f7a-4793-a724-9fd5022740d7/console-operator/1.log"
Sep 18 13:15:41.390293 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:41.390273    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k86vn_55e33e14-3f7a-4793-a724-9fd5022740d7/console-operator/0.log"
Sep 18 13:15:41.390349 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:41.390315    2566 generic.go:356] "Generic (PLEG): container finished" podID="55e33e14-3f7a-4793-a724-9fd5022740d7" containerID="e92439c1dbd6073e87b328705fe59c2377011692dbddb1267de8cb94ac341651" exitCode=255
Sep 18 13:15:41.390390 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:41.390357    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-k86vn" event={"ID":"55e33e14-3f7a-4793-a724-9fd5022740d7","Type":"ContainerDied","Data":"e92439c1dbd6073e87b328705fe59c2377011692dbddb1267de8cb94ac341651"}
Sep 18 13:15:41.390468 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:41.390448    2566 scope.go:117] "RemoveContainer" containerID="36e9e0ebae15cc1952311bd883a581bb9a701df2a5cc1b732f7fc4b07f1dc50a"
Sep 18 13:15:41.390674 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:41.390659    2566 scope.go:117] "RemoveContainer" containerID="e92439c1dbd6073e87b328705fe59c2377011692dbddb1267de8cb94ac341651"
Sep 18 13:15:41.390865 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:41.390843    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-k86vn_openshift-console-operator(55e33e14-3f7a-4793-a724-9fd5022740d7)\"" pod="openshift-console-operator/console-operator-fd4bb46c-k86vn" podUID="55e33e14-3f7a-4793-a724-9fd5022740d7"
Sep 18 13:15:42.339478 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:42.339437    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96cd73fd-fcb1-44ea-b184-63a97a0277f4-metrics-tls\") pod \"dns-default-r94bv\" (UID: \"96cd73fd-fcb1-44ea-b184-63a97a0277f4\") " pod="openshift-dns/dns-default-r94bv"
Sep 18 13:15:42.339684 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:42.339609    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:15:42.339684 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:42.339679    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96cd73fd-fcb1-44ea-b184-63a97a0277f4-metrics-tls podName:96cd73fd-fcb1-44ea-b184-63a97a0277f4 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:58.339660771 +0000 UTC m=+66.774537153 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96cd73fd-fcb1-44ea-b184-63a97a0277f4-metrics-tls") pod "dns-default-r94bv" (UID: "96cd73fd-fcb1-44ea-b184-63a97a0277f4") : secret "dns-default-metrics-tls" not found
Sep 18 13:15:42.394756 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:42.394730    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k86vn_55e33e14-3f7a-4793-a724-9fd5022740d7/console-operator/1.log"
Sep 18 13:15:42.395120 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:42.395039    2566 scope.go:117] "RemoveContainer" containerID="e92439c1dbd6073e87b328705fe59c2377011692dbddb1267de8cb94ac341651"
Sep 18 13:15:42.395201 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:42.395181    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-k86vn_openshift-console-operator(55e33e14-3f7a-4793-a724-9fd5022740d7)\"" pod="openshift-console-operator/console-operator-fd4bb46c-k86vn" podUID="55e33e14-3f7a-4793-a724-9fd5022740d7"
Sep 18 13:15:43.670565 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:43.670526    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-k86vn"
Sep 18 13:15:43.670941 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:43.670630    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-k86vn"
Sep 18 13:15:43.670995 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:43.670943    2566 scope.go:117] "RemoveContainer" containerID="e92439c1dbd6073e87b328705fe59c2377011692dbddb1267de8cb94ac341651"
Sep 18 13:15:43.671101 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:43.671082    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-k86vn_openshift-console-operator(55e33e14-3f7a-4793-a724-9fd5022740d7)\"" pod="openshift-console-operator/console-operator-fd4bb46c-k86vn" podUID="55e33e14-3f7a-4793-a724-9fd5022740d7"
Sep 18 13:15:45.061704 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:45.061657    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d630003-2bfd-4cb9-9b8a-745c0ccca548-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:45.062126 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:45.061747    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d630003-2bfd-4cb9-9b8a-745c0ccca548-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:45.062126 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:45.061807    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:15:45.062126 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:45.061863    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:15:45.062126 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:45.061876    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d630003-2bfd-4cb9-9b8a-745c0ccca548-insights-runtime-extractor-tls podName:3d630003-2bfd-4cb9-9b8a-745c0ccca548 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:53.061861473 +0000 UTC m=+61.496737855 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d630003-2bfd-4cb9-9b8a-745c0ccca548-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xjrc4" (UID: "3d630003-2bfd-4cb9-9b8a-745c0ccca548") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:15:45.062126 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:45.061912    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d630003-2bfd-4cb9-9b8a-745c0ccca548-kube-rbac-proxy-cm podName:3d630003-2bfd-4cb9-9b8a-745c0ccca548 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:53.061899225 +0000 UTC m=+61.496775607 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3d630003-2bfd-4cb9-9b8a-745c0ccca548-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xjrc4" (UID: "3d630003-2bfd-4cb9-9b8a-745c0ccca548") : configmap "kube-rbac-proxy" not found
Sep 18 13:15:48.893168 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:48.893127    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b914660-2e06-4c5f-897f-f3ac5b12c348-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-pg4dm\" (UID: \"6b914660-2e06-4c5f-897f-f3ac5b12c348\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pg4dm"
Sep 18 13:15:48.893722 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:48.893247    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 13:15:48.893722 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:48.893301    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b914660-2e06-4c5f-897f-f3ac5b12c348-cluster-monitoring-operator-tls podName:6b914660-2e06-4c5f-897f-f3ac5b12c348 nodeName:}" failed. No retries permitted until 2026-09-18 13:16:04.893288556 +0000 UTC m=+73.328164937 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b914660-2e06-4c5f-897f-f3ac5b12c348-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-pg4dm" (UID: "6b914660-2e06-4c5f-897f-f3ac5b12c348") : secret "cluster-monitoring-operator-tls" not found
Sep 18 13:15:48.993476 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:48.993433    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-ggbng\" (UID: \"b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-ggbng"
Sep 18 13:15:48.995765 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:48.995736    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-ggbng\" (UID: \"b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-ggbng"
Sep 18 13:15:49.089871 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:49.089837    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-ggbng"
Sep 18 13:15:49.296252 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:49.296217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-tls\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:15:49.296407 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:49.296364    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:15:49.296407 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:49.296382    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668879999c-dfvz7: secret "image-registry-tls" not found
Sep 18 13:15:49.296475 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:49.296438    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-tls podName:0bb07b38-9199-424d-9fb7-3ffa57def30a nodeName:}" failed. No retries permitted until 2026-09-18 13:16:05.296423099 +0000 UTC m=+73.731299481 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-tls") pod "image-registry-668879999c-dfvz7" (UID: "0bb07b38-9199-424d-9fb7-3ffa57def30a") : secret "image-registry-tls" not found
Sep 18 13:15:49.394607 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:49.394570    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-ggbng"]
Sep 18 13:15:50.420434 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:50.420402    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-ggbng" event={"ID":"b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0","Type":"ContainerStarted","Data":"ba71f89300e255daadcc818d77780cce50f928cf21a563ff8f79603dd76417d5"}
Sep 18 13:15:51.424811 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:51.424776    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-ggbng" event={"ID":"b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0","Type":"ContainerStarted","Data":"a9990beabb23e1375a954c22350115021f960685b2ce90e2b9fe554119b5333f"}
Sep 18 13:15:51.425278 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:51.424835    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-ggbng" event={"ID":"b0a9d6b9-4fe5-4a87-87b3-0a5f27b626f0","Type":"ContainerStarted","Data":"7a7d9b7e29337910ad43376fa964b1068bbf99d84777166ee5207737bc2a1671"}
Sep 18 13:15:52.321400 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:52.321361    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ljpp4"
Sep 18 13:15:52.382471 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:52.382415    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-ggbng" podStartSLOduration=17.759772058 podStartE2EDuration="19.382399595s" totalImagesPullingTime="1.622627537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:51.459850214 +0000 UTC m=+59.894726620" watchObservedRunningTime="2026-09-18 13:15:52.382399595 +0000 UTC m=+60.817275999"
Sep 18 13:15:53.127581 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:53.127523    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d630003-2bfd-4cb9-9b8a-745c0ccca548-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:53.127581 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:53.127615    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d630003-2bfd-4cb9-9b8a-745c0ccca548-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:53.128049 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:53.127685    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:15:53.128049 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:15:53.127755    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d630003-2bfd-4cb9-9b8a-745c0ccca548-insights-runtime-extractor-tls podName:3d630003-2bfd-4cb9-9b8a-745c0ccca548 nodeName:}" failed. No retries permitted until 2026-09-18 13:16:09.127740266 +0000 UTC m=+77.562616652 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d630003-2bfd-4cb9-9b8a-745c0ccca548-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xjrc4" (UID: "3d630003-2bfd-4cb9-9b8a-745c0ccca548") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:15:53.128129 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:53.128089    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d630003-2bfd-4cb9-9b8a-745c0ccca548-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:15:57.071010 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:57.070971    2566 scope.go:117] "RemoveContainer" containerID="e92439c1dbd6073e87b328705fe59c2377011692dbddb1267de8cb94ac341651"
Sep 18 13:15:57.444506 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:57.444480    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k86vn_55e33e14-3f7a-4793-a724-9fd5022740d7/console-operator/1.log"
Sep 18 13:15:57.444696 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:57.444539    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-k86vn" event={"ID":"55e33e14-3f7a-4793-a724-9fd5022740d7","Type":"ContainerStarted","Data":"7a3b685de3af7dca416f7e3d53760607d861b2d8f9b466c63328121381cf9c8e"}
Sep 18 13:15:57.444861 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:57.444826    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-k86vn"
Sep 18 13:15:57.469919 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:57.469863    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-k86vn" podStartSLOduration=19.070491826 podStartE2EDuration="24.469848902s" totalImagesPullingTime="5.399357076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:57.469220275 +0000 UTC m=+65.904096673" watchObservedRunningTime="2026-09-18 13:15:57.469848902 +0000 UTC m=+65.904725298"
Sep 18 13:15:57.641485 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:57.641451    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-k86vn"
Sep 18 13:15:57.866295 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:57.866262    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqn77\" (UniqueName: \"kubernetes.io/projected/56a3b9a4-38a4-4412-9876-95c3c51c8aff-kube-api-access-lqn77\") pod \"network-check-target-vdj8k\" (UID: \"56a3b9a4-38a4-4412-9876-95c3c51c8aff\") " pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:15:57.866477 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:57.866347    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7608b5b-a581-407b-94b9-3bd63a76c9f0-metrics-certs\") pod \"network-metrics-daemon-h9kt5\" (UID: \"e7608b5b-a581-407b-94b9-3bd63a76c9f0\") " pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:15:57.866477 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:57.866367    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15a0691d-d5b4-41d9-a89c-705ee52f7b25-original-pull-secret\") pod \"global-pull-secret-syncer-bgp65\" (UID: \"15a0691d-d5b4-41d9-a89c-705ee52f7b25\") " pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:15:57.868745 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:57.868723    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqn77\" (UniqueName: \"kubernetes.io/projected/56a3b9a4-38a4-4412-9876-95c3c51c8aff-kube-api-access-lqn77\") pod \"network-check-target-vdj8k\" (UID: \"56a3b9a4-38a4-4412-9876-95c3c51c8aff\") " pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:15:57.873176 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:57.873156    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 13:15:57.873439 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:57.873420    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 13:15:57.878813 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:57.878786    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/15a0691d-d5b4-41d9-a89c-705ee52f7b25-original-pull-secret\") pod \"global-pull-secret-syncer-bgp65\" (UID: \"15a0691d-d5b4-41d9-a89c-705ee52f7b25\") " pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:15:57.878915 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:57.878794    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7608b5b-a581-407b-94b9-3bd63a76c9f0-metrics-certs\") pod \"network-metrics-daemon-h9kt5\" (UID: \"e7608b5b-a581-407b-94b9-3bd63a76c9f0\") " pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:15:58.086215 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:58.086186    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4rmbt\""
Sep 18 13:15:58.090746 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:58.090728    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bgp65"
Sep 18 13:15:58.094419 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:58.093858    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h9kt5"
Sep 18 13:15:58.099834 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:58.099805    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-crtpr\""
Sep 18 13:15:58.106932 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:58.106905    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:15:58.370124 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:58.370091    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96cd73fd-fcb1-44ea-b184-63a97a0277f4-metrics-tls\") pod \"dns-default-r94bv\" (UID: \"96cd73fd-fcb1-44ea-b184-63a97a0277f4\") " pod="openshift-dns/dns-default-r94bv"
Sep 18 13:15:58.372354 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:58.372336    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96cd73fd-fcb1-44ea-b184-63a97a0277f4-metrics-tls\") pod \"dns-default-r94bv\" (UID: \"96cd73fd-fcb1-44ea-b184-63a97a0277f4\") " pod="openshift-dns/dns-default-r94bv"
Sep 18 13:15:58.390233 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:58.390209    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bgp65"]
Sep 18 13:15:58.392413 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:58.392392    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h9kt5"]
Sep 18 13:15:58.394077 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:15:58.394046    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a0691d_d5b4_41d9_a89c_705ee52f7b25.slice/crio-c01eeeb7ce51baa703cfc8143d33d2f90c1f76015f889bf57453823b2ff077c3 WatchSource:0}: Error finding container c01eeeb7ce51baa703cfc8143d33d2f90c1f76015f889bf57453823b2ff077c3: Status 404 returned error can't find the container with id c01eeeb7ce51baa703cfc8143d33d2f90c1f76015f889bf57453823b2ff077c3
Sep 18 13:15:58.395317 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:15:58.395297    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7608b5b_a581_407b_94b9_3bd63a76c9f0.slice/crio-dbf0a1740f5692f2bfb5d7e4b1e7a90ae61a654ccc18b458d65e62c832e1b7cd WatchSource:0}: Error finding container dbf0a1740f5692f2bfb5d7e4b1e7a90ae61a654ccc18b458d65e62c832e1b7cd: Status 404 returned error can't find the container with id dbf0a1740f5692f2bfb5d7e4b1e7a90ae61a654ccc18b458d65e62c832e1b7cd
Sep 18 13:15:58.424259 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:58.424207    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vdj8k"]
Sep 18 13:15:58.426969 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:15:58.426947    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a3b9a4_38a4_4412_9876_95c3c51c8aff.slice/crio-7454c591d504df682d5a86318f38c4db1665adc77ccbf6538d6c8cc24774f149 WatchSource:0}: Error finding container 7454c591d504df682d5a86318f38c4db1665adc77ccbf6538d6c8cc24774f149: Status 404 returned error can't find the container with id 7454c591d504df682d5a86318f38c4db1665adc77ccbf6538d6c8cc24774f149
Sep 18 13:15:58.449123 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:58.449097    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vdj8k" event={"ID":"56a3b9a4-38a4-4412-9876-95c3c51c8aff","Type":"ContainerStarted","Data":"7454c591d504df682d5a86318f38c4db1665adc77ccbf6538d6c8cc24774f149"}
Sep 18 13:15:58.450294 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:58.450267    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h9kt5" event={"ID":"e7608b5b-a581-407b-94b9-3bd63a76c9f0","Type":"ContainerStarted","Data":"dbf0a1740f5692f2bfb5d7e4b1e7a90ae61a654ccc18b458d65e62c832e1b7cd"}
Sep 18 13:15:58.451431 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:58.451413    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bgp65" event={"ID":"15a0691d-d5b4-41d9-a89c-705ee52f7b25","Type":"ContainerStarted","Data":"c01eeeb7ce51baa703cfc8143d33d2f90c1f76015f889bf57453823b2ff077c3"}
Sep 18 13:15:58.581141 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:58.581107    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5qlvl\""
Sep 18 13:15:58.589355 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:58.589334    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r94bv"
Sep 18 13:15:58.869886 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:58.869852    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r94bv"]
Sep 18 13:15:58.875540 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:15:58.875508    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96cd73fd_fcb1_44ea_b184_63a97a0277f4.slice/crio-2b2a89a1e4a58ee8df5179310badd6b78118ccf30f8d823496d001e5d541ac63 WatchSource:0}: Error finding container 2b2a89a1e4a58ee8df5179310badd6b78118ccf30f8d823496d001e5d541ac63: Status 404 returned error can't find the container with id 2b2a89a1e4a58ee8df5179310badd6b78118ccf30f8d823496d001e5d541ac63
Sep 18 13:15:59.457866 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:59.457829    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vdj8k" event={"ID":"56a3b9a4-38a4-4412-9876-95c3c51c8aff","Type":"ContainerStarted","Data":"2e1e5b8ba53aa0597d0e5fc022065ee8fcd5bd6546bf3b8ce3b8f7f9d201a65f"}
Sep 18 13:15:59.458605 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:59.458571    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:15:59.460294 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:15:59.460229    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r94bv" event={"ID":"96cd73fd-fcb1-44ea-b184-63a97a0277f4","Type":"ContainerStarted","Data":"2b2a89a1e4a58ee8df5179310badd6b78118ccf30f8d823496d001e5d541ac63"}
Sep 18 13:16:00.467881 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:00.467841    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h9kt5" event={"ID":"e7608b5b-a581-407b-94b9-3bd63a76c9f0","Type":"ContainerStarted","Data":"3db151689031349f2605ad0991377800a895d19905bf46543155149ed805cb5b"}
Sep 18 13:16:00.468343 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:00.467908    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h9kt5" event={"ID":"e7608b5b-a581-407b-94b9-3bd63a76c9f0","Type":"ContainerStarted","Data":"480498f5351c02ecfe41730b67480655e60997c648b069c4a79c5d87b51f795a"}
Sep 18 13:16:00.502361 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:00.502299    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-h9kt5" podStartSLOduration=66.538623804 podStartE2EDuration="1m7.502280402s" totalImagesPullingTime="963.656598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:16:00.501895605 +0000 UTC m=+68.936772010" watchObservedRunningTime="2026-09-18 13:16:00.502280402 +0000 UTC m=+68.937156836"
Sep 18 13:16:00.502540 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:00.502410    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vdj8k" podStartSLOduration=68.502402226 podStartE2EDuration="1m8.502402226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:59.503733453 +0000 UTC m=+67.938609858" watchObservedRunningTime="2026-09-18 13:16:00.502402226 +0000 UTC m=+68.937278630"
Sep 18 13:16:01.473106 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:01.473067    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r94bv" event={"ID":"96cd73fd-fcb1-44ea-b184-63a97a0277f4","Type":"ContainerStarted","Data":"5bca59d5841f53826d264cad0524f8418e2a9ec4814c7bf42135b3061e3b5484"}
Sep 18 13:16:02.479835 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:02.479739    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bgp65" event={"ID":"15a0691d-d5b4-41d9-a89c-705ee52f7b25","Type":"ContainerStarted","Data":"9250928e49014e8a75ef464e69e5b92c2f70829fe5c58ebcd1a5d94583d9699e"}
Sep 18 13:16:02.481532 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:02.481507    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r94bv" event={"ID":"96cd73fd-fcb1-44ea-b184-63a97a0277f4","Type":"ContainerStarted","Data":"2d63784afbe09be81d7ed01dc47cc33e8db48bde521cf716dba80b13e2d9a000"}
Sep 18 13:16:02.481683 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:02.481551    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r94bv"
Sep 18 13:16:02.499365 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:02.499314    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bgp65" podStartSLOduration=65.70130989 podStartE2EDuration="1m9.499295509s" totalImagesPullingTime="3.797985619s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:16:02.499146002 +0000 UTC m=+70.934022407" watchObservedRunningTime="2026-09-18 13:16:02.499295509 +0000 UTC m=+70.934171916"
Sep 18 13:16:04.927746 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:04.927708    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b914660-2e06-4c5f-897f-f3ac5b12c348-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-pg4dm\" (UID: \"6b914660-2e06-4c5f-897f-f3ac5b12c348\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pg4dm"
Sep 18 13:16:04.930073 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:04.930049    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b914660-2e06-4c5f-897f-f3ac5b12c348-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-pg4dm\" (UID: \"6b914660-2e06-4c5f-897f-f3ac5b12c348\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pg4dm"
Sep 18 13:16:05.181269 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:05.181185    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-68zdw\""
Sep 18 13:16:05.189139 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:05.189115    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pg4dm"
Sep 18 13:16:05.330637 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:05.330579    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-tls\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:16:05.332948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:05.332927    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-tls\") pod \"image-registry-668879999c-dfvz7\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") " pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:16:05.460655 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:05.460534    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-r94bv" podStartSLOduration=37.585582281 podStartE2EDuration="39.460515355s" totalImagesPullingTime="1.874933074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:16:02.524785885 +0000 UTC m=+70.959662290" watchObservedRunningTime="2026-09-18 13:16:05.460515355 +0000 UTC m=+73.895391760"
Sep 18 13:16:05.461033 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:05.461015    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pg4dm"]
Sep 18 13:16:05.464280 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:16:05.464257    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b914660_2e06_4c5f_897f_f3ac5b12c348.slice/crio-6212b5dd9d35a49ef2e74fec75c3bb270c31edb9cb6c3373c1e6c3108a3272c7 WatchSource:0}: Error finding container 6212b5dd9d35a49ef2e74fec75c3bb270c31edb9cb6c3373c1e6c3108a3272c7: Status 404 returned error can't find the container with id 6212b5dd9d35a49ef2e74fec75c3bb270c31edb9cb6c3373c1e6c3108a3272c7
Sep 18 13:16:05.493204 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:05.493178    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pg4dm" event={"ID":"6b914660-2e06-4c5f-897f-f3ac5b12c348","Type":"ContainerStarted","Data":"6212b5dd9d35a49ef2e74fec75c3bb270c31edb9cb6c3373c1e6c3108a3272c7"}
Sep 18 13:16:05.549929 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:05.549902    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w5nbx\""
Sep 18 13:16:05.557783 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:05.557759    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:16:05.836315 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:05.836241    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-668879999c-dfvz7"]
Sep 18 13:16:05.839123 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:16:05.839099    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb07b38_9199_424d_9fb7_3ffa57def30a.slice/crio-1c3df0b02a0878264582aab2a5be2b04e2fbe7b4fda906ad6f163958dac76c4e WatchSource:0}: Error finding container 1c3df0b02a0878264582aab2a5be2b04e2fbe7b4fda906ad6f163958dac76c4e: Status 404 returned error can't find the container with id 1c3df0b02a0878264582aab2a5be2b04e2fbe7b4fda906ad6f163958dac76c4e
Sep 18 13:16:06.499159 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:06.499119    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-668879999c-dfvz7" event={"ID":"0bb07b38-9199-424d-9fb7-3ffa57def30a","Type":"ContainerStarted","Data":"9a4737325d61cfe2f696084433aab8569b9837522063c68f0433176990426736"}
Sep 18 13:16:06.499572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:06.499183    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-668879999c-dfvz7" event={"ID":"0bb07b38-9199-424d-9fb7-3ffa57def30a","Type":"ContainerStarted","Data":"1c3df0b02a0878264582aab2a5be2b04e2fbe7b4fda906ad6f163958dac76c4e"}
Sep 18 13:16:06.499572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:06.499250    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:16:06.520248 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:06.520199    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-668879999c-dfvz7" podStartSLOduration=33.520183013 podStartE2EDuration="33.520183013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:16:06.519714974 +0000 UTC m=+74.954591379" watchObservedRunningTime="2026-09-18 13:16:06.520183013 +0000 UTC m=+74.955059418"
Sep 18 13:16:07.505011 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:07.504903    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pg4dm" event={"ID":"6b914660-2e06-4c5f-897f-f3ac5b12c348","Type":"ContainerStarted","Data":"f5a039f2eebd4a030306b75b210ad9cda9a26ded4b29de1d6624154c416a8f4b"}
Sep 18 13:16:07.527492 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:07.527434    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-pg4dm" podStartSLOduration=33.851971943 podStartE2EDuration="35.52741348s" totalImagesPullingTime="1.675441537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:16:07.526562977 +0000 UTC m=+75.961439384" watchObservedRunningTime="2026-09-18 13:16:07.52741348 +0000 UTC m=+75.962289885"
Sep 18 13:16:09.160838 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:09.160803    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d630003-2bfd-4cb9-9b8a-745c0ccca548-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:16:09.163232 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:09.163207    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d630003-2bfd-4cb9-9b8a-745c0ccca548-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xjrc4\" (UID: \"3d630003-2bfd-4cb9-9b8a-745c0ccca548\") " pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:16:09.351497 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:09.351469    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p2lqc\""
Sep 18 13:16:09.360337 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:09.360318    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xjrc4"
Sep 18 13:16:09.652141 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:09.652106    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xjrc4"]
Sep 18 13:16:09.655188 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:16:09.655160    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d630003_2bfd_4cb9_9b8a_745c0ccca548.slice/crio-03d8364fda6d4336c3dbf96be6b252e2c1e86e111ab058826e8cfa8474c355fd WatchSource:0}: Error finding container 03d8364fda6d4336c3dbf96be6b252e2c1e86e111ab058826e8cfa8474c355fd: Status 404 returned error can't find the container with id 03d8364fda6d4336c3dbf96be6b252e2c1e86e111ab058826e8cfa8474c355fd
Sep 18 13:16:10.517392 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:10.517359    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xjrc4" event={"ID":"3d630003-2bfd-4cb9-9b8a-745c0ccca548","Type":"ContainerStarted","Data":"a0248fdd1fccf54eb0c56d7d09568054e37bbe6e3bd5669db8858a9891b8364a"}
Sep 18 13:16:10.517787 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:10.517413    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xjrc4" event={"ID":"3d630003-2bfd-4cb9-9b8a-745c0ccca548","Type":"ContainerStarted","Data":"03d8364fda6d4336c3dbf96be6b252e2c1e86e111ab058826e8cfa8474c355fd"}
Sep 18 13:16:10.902337 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:10.902304    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69dc79fc8-8fgtc"]
Sep 18 13:16:10.925173 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:10.925149    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69dc79fc8-8fgtc"]
Sep 18 13:16:10.925300 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:10.925259    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69dc79fc8-8fgtc"
Sep 18 13:16:10.931421 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:10.931384    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 13:16:10.931421 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:10.931393    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 13:16:10.932062 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:10.932042    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 13:16:10.937885 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:10.937870    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 18 13:16:10.974704 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:10.974674    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/aa7648d3-5933-464c-a528-19a9dc414e34-klusterlet-config\") pod \"klusterlet-addon-workmgr-69dc79fc8-8fgtc\" (UID: \"aa7648d3-5933-464c-a528-19a9dc414e34\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69dc79fc8-8fgtc"
Sep 18 13:16:10.974857 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:10.974718    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgv4r\" (UniqueName: \"kubernetes.io/projected/aa7648d3-5933-464c-a528-19a9dc414e34-kube-api-access-hgv4r\") pod \"klusterlet-addon-workmgr-69dc79fc8-8fgtc\" (UID: \"aa7648d3-5933-464c-a528-19a9dc414e34\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69dc79fc8-8fgtc"
Sep 18 13:16:10.974857 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:10.974820    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa7648d3-5933-464c-a528-19a9dc414e34-tmp\") pod \"klusterlet-addon-workmgr-69dc79fc8-8fgtc\" (UID: \"aa7648d3-5933-464c-a528-19a9dc414e34\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69dc79fc8-8fgtc"
Sep 18 13:16:11.026904 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:11.026876    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-vtcgz"]
Sep 18 13:16:11.076136 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:11.076022    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa7648d3-5933-464c-a528-19a9dc414e34-tmp\") pod \"klusterlet-addon-workmgr-69dc79fc8-8fgtc\" (UID: \"aa7648d3-5933-464c-a528-19a9dc414e34\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69dc79fc8-8fgtc"
Sep 18 13:16:11.076136 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:11.076067    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/aa7648d3-5933-464c-a528-19a9dc414e34-klusterlet-config\") pod \"klusterlet-addon-workmgr-69dc79fc8-8fgtc\" (UID: \"aa7648d3-5933-464c-a528-19a9dc414e34\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69dc79fc8-8fgtc"
Sep 18 13:16:11.076136 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:11.076091    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgv4r\" (UniqueName: \"kubernetes.io/projected/aa7648d3-5933-464c-a528-19a9dc414e34-kube-api-access-hgv4r\") pod \"klusterlet-addon-workmgr-69dc79fc8-8fgtc\" (UID: \"aa7648d3-5933-464c-a528-19a9dc414e34\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69dc79fc8-8fgtc"
Sep 18 13:16:11.076501 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:11.076478    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa7648d3-5933-464c-a528-19a9dc414e34-tmp\") pod \"klusterlet-addon-workmgr-69dc79fc8-8fgtc\" (UID: \"aa7648d3-5933-464c-a528-19a9dc414e34\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69dc79fc8-8fgtc"
Sep 18 13:16:11.078567 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:11.078545    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/aa7648d3-5933-464c-a528-19a9dc414e34-klusterlet-config\") pod \"klusterlet-addon-workmgr-69dc79fc8-8fgtc\" (UID: \"aa7648d3-5933-464c-a528-19a9dc414e34\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69dc79fc8-8fgtc"
Sep 18 13:16:11.091029 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:11.090992    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-668879999c-dfvz7"]
Sep 18 13:16:11.091137 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:11.091044    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-vtcgz"]
Sep 18 13:16:11.091137 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:11.091116    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-vtcgz"
Sep 18 13:16:11.094150 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:11.094127    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-zpqwt\""
Sep 18 13:16:11.094275 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:11.094151    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 13:16:11.094463 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:11.094444    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 13:16:11.096276 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:11.096251    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgv4r\" (UniqueName: \"kubernetes.io/projected/aa7648d3-5933-464c-a528-19a9dc414e34-kube-api-access-hgv4r\") pod \"klusterlet-addon-workmgr-69dc79fc8-8fgtc\" (UID: \"aa7648d3-5933-464c-a528-19a9dc414e34\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69dc79fc8-8fgtc"
Sep 18 13:16:11.177389 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:11.177353    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45fj5\" (UniqueName: \"kubernetes.io/projected/e167d2a4-7a91-46b1-a5e3-da0a27ede351-kube-api-access-45fj5\") pod \"downloads-6594f4c75-vtcgz\" (UID: \"e167d2a4-7a91-46b1-a5e3-da0a27ede351\") " pod="openshift-console/downloads-6594f4c75-vtcgz"
Sep 18 13:16:11.235867 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:11.235828    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69dc79fc8-8fgtc"
Sep 18 13:16:11.277955 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:11.277930    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45fj5\" (UniqueName: \"kubernetes.io/projected/e167d2a4-7a91-46b1-a5e3-da0a27ede351-kube-api-access-45fj5\") pod \"downloads-6594f4c75-vtcgz\" (UID: \"e167d2a4-7a91-46b1-a5e3-da0a27ede351\") " pod="openshift-console/downloads-6594f4c75-vtcgz"
Sep 18 13:16:11.288982 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:11.288943    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45fj5\" (UniqueName: \"kubernetes.io/projected/e167d2a4-7a91-46b1-a5e3-da0a27ede351-kube-api-access-45fj5\") pod \"downloads-6594f4c75-vtcgz\" (UID: \"e167d2a4-7a91-46b1-a5e3-da0a27ede351\") " pod="openshift-console/downloads-6594f4c75-vtcgz"
Sep 18 13:16:11.406639 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:11.406577    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-vtcgz"
Sep 18 13:16:11.524043 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:11.524008    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xjrc4" event={"ID":"3d630003-2bfd-4cb9-9b8a-745c0ccca548","Type":"ContainerStarted","Data":"89e0ccf8008fbf7ed3ba65ebe612b98d45d27dc414e6f80ab9904ac4d2f776f2"}
Sep 18 13:16:11.539864 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:11.539838    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69dc79fc8-8fgtc"]
Sep 18 13:16:11.542824 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:16:11.542789    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa7648d3_5933_464c_a528_19a9dc414e34.slice/crio-02d5ccae85780089f59184bdeeaba5e9c070f0b8d1b2b026db6e6185294f659a WatchSource:0}: Error finding container 02d5ccae85780089f59184bdeeaba5e9c070f0b8d1b2b026db6e6185294f659a: Status 404 returned error can't find the container with id 02d5ccae85780089f59184bdeeaba5e9c070f0b8d1b2b026db6e6185294f659a
Sep 18 13:16:11.706726 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:11.706581    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-vtcgz"]
Sep 18 13:16:11.711867 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:16:11.711802    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode167d2a4_7a91_46b1_a5e3_da0a27ede351.slice/crio-ee8946a5c6982922e8a1d78487628833d3b90884ee5944fcf06c0453dae3dd4a WatchSource:0}: Error finding container ee8946a5c6982922e8a1d78487628833d3b90884ee5944fcf06c0453dae3dd4a: Status 404 returned error can't find the container with id ee8946a5c6982922e8a1d78487628833d3b90884ee5944fcf06c0453dae3dd4a
Sep 18 13:16:12.487857 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:12.487824    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r94bv"
Sep 18 13:16:12.532348 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:12.532226    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-vtcgz" event={"ID":"e167d2a4-7a91-46b1-a5e3-da0a27ede351","Type":"ContainerStarted","Data":"ee8946a5c6982922e8a1d78487628833d3b90884ee5944fcf06c0453dae3dd4a"}
Sep 18 13:16:12.535713 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:12.535634    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xjrc4" event={"ID":"3d630003-2bfd-4cb9-9b8a-745c0ccca548","Type":"ContainerStarted","Data":"25f209a4a27e9a398a5801e42f6ea437e60fa2f053cc4b74aec9dead37ba6f88"}
Sep 18 13:16:12.539270 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:12.539207    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69dc79fc8-8fgtc" event={"ID":"aa7648d3-5933-464c-a528-19a9dc414e34","Type":"ContainerStarted","Data":"02d5ccae85780089f59184bdeeaba5e9c070f0b8d1b2b026db6e6185294f659a"}
Sep 18 13:16:12.557260 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:12.556849    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xjrc4" podStartSLOduration=33.087357487 podStartE2EDuration="35.55682731s" totalImagesPullingTime="2.469469823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:15:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:16:12.555107968 +0000 UTC m=+80.989984374" watchObservedRunningTime="2026-09-18 13:16:12.55682731 +0000 UTC m=+80.991703715"
Sep 18 13:16:16.558535 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:16.558494    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69dc79fc8-8fgtc" event={"ID":"aa7648d3-5933-464c-a528-19a9dc414e34","Type":"ContainerStarted","Data":"8082c35818639a489f290356edb4b048827e495be61aa2130536408ebef97949"}
Sep 18 13:16:16.559045 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:16.558715    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69dc79fc8-8fgtc"
Sep 18 13:16:16.560430 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:16.560408    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69dc79fc8-8fgtc"
Sep 18 13:16:16.586480 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:16.586420    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69dc79fc8-8fgtc" podStartSLOduration=2.216123829 podStartE2EDuration="6.586399265s" totalImagesPullingTime="4.370275436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:16:16.585375296 +0000 UTC m=+85.020251701" watchObservedRunningTime="2026-09-18 13:16:16.586399265 +0000 UTC m=+85.021275670"
Sep 18 13:16:21.098456 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:21.098423    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:16:23.190096 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.190044    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-9d24f"]
Sep 18 13:16:23.196764 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.196741    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f"
Sep 18 13:16:23.199075 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.199050    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 13:16:23.199618 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.199573    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 18 13:16:23.199726 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.199691    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 18 13:16:23.199726 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.199711    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-fkdsz\""
Sep 18 13:16:23.199961 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.199942    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 18 13:16:23.202476 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.202455    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tf756"]
Sep 18 13:16:23.206777 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.206746    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.209547 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.209525    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 13:16:23.209835 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.209817    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 13:16:23.209946 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.209924    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 13:16:23.210291 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.210273    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p6v7j\""
Sep 18 13:16:23.220749 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.220726    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-9d24f"]
Sep 18 13:16:23.281063 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.281022    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdc18c51-ef7d-45ee-9e5c-7805dfcf618c-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-9d24f\" (UID: \"bdc18c51-ef7d-45ee-9e5c-7805dfcf618c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f"
Sep 18 13:16:23.281248 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.281074    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bdc18c51-ef7d-45ee-9e5c-7805dfcf618c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-9d24f\" (UID: \"bdc18c51-ef7d-45ee-9e5c-7805dfcf618c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f"
Sep 18 13:16:23.281248 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.281135    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bdc18c51-ef7d-45ee-9e5c-7805dfcf618c-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-9d24f\" (UID: \"bdc18c51-ef7d-45ee-9e5c-7805dfcf618c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f"
Sep 18 13:16:23.281248 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.281191    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bdc18c51-ef7d-45ee-9e5c-7805dfcf618c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-9d24f\" (UID: \"bdc18c51-ef7d-45ee-9e5c-7805dfcf618c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f"
Sep 18 13:16:23.281248 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.281220    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq4jm\" (UniqueName: \"kubernetes.io/projected/bdc18c51-ef7d-45ee-9e5c-7805dfcf618c-kube-api-access-nq4jm\") pod \"kube-state-metrics-84c885658c-9d24f\" (UID: \"bdc18c51-ef7d-45ee-9e5c-7805dfcf618c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f"
Sep 18 13:16:23.281248 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.281247    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdc18c51-ef7d-45ee-9e5c-7805dfcf618c-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-9d24f\" (UID: \"bdc18c51-ef7d-45ee-9e5c-7805dfcf618c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f"
Sep 18 13:16:23.382020 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.381982    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bdc18c51-ef7d-45ee-9e5c-7805dfcf618c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-9d24f\" (UID: \"bdc18c51-ef7d-45ee-9e5c-7805dfcf618c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f"
Sep 18 13:16:23.382209 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.382029    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zw6m\" (UniqueName: \"kubernetes.io/projected/6c168813-ac20-48b4-b2a3-48664b28466a-kube-api-access-2zw6m\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.382209 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.382061    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq4jm\" (UniqueName: \"kubernetes.io/projected/bdc18c51-ef7d-45ee-9e5c-7805dfcf618c-kube-api-access-nq4jm\") pod \"kube-state-metrics-84c885658c-9d24f\" (UID: \"bdc18c51-ef7d-45ee-9e5c-7805dfcf618c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f"
Sep 18 13:16:23.382209 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.382141    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdc18c51-ef7d-45ee-9e5c-7805dfcf618c-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-9d24f\" (UID: \"bdc18c51-ef7d-45ee-9e5c-7805dfcf618c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f"
Sep 18 13:16:23.382209 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.382191    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6c168813-ac20-48b4-b2a3-48664b28466a-node-exporter-textfile\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.382426 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.382307    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6c168813-ac20-48b4-b2a3-48664b28466a-root\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.382426 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.382357    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdc18c51-ef7d-45ee-9e5c-7805dfcf618c-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-9d24f\" (UID: \"bdc18c51-ef7d-45ee-9e5c-7805dfcf618c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f"
Sep 18 13:16:23.382426 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.382387    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6c168813-ac20-48b4-b2a3-48664b28466a-node-exporter-wtmp\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.382426 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.382417    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6c168813-ac20-48b4-b2a3-48664b28466a-node-exporter-tls\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.382635 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.382442    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bdc18c51-ef7d-45ee-9e5c-7805dfcf618c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-9d24f\" (UID: \"bdc18c51-ef7d-45ee-9e5c-7805dfcf618c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f"
Sep 18 13:16:23.382635 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.382463    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c168813-ac20-48b4-b2a3-48664b28466a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.382635 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.382499    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bdc18c51-ef7d-45ee-9e5c-7805dfcf618c-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-9d24f\" (UID: \"bdc18c51-ef7d-45ee-9e5c-7805dfcf618c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f"
Sep 18 13:16:23.382795 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.382655    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c168813-ac20-48b4-b2a3-48664b28466a-sys\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.382795 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.382702    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6c168813-ac20-48b4-b2a3-48664b28466a-node-exporter-accelerators-collector-config\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.382795 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.382728    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c168813-ac20-48b4-b2a3-48664b28466a-metrics-client-ca\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.382955 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.382806    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bdc18c51-ef7d-45ee-9e5c-7805dfcf618c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-9d24f\" (UID: \"bdc18c51-ef7d-45ee-9e5c-7805dfcf618c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f"
Sep 18 13:16:23.382955 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.382852    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bdc18c51-ef7d-45ee-9e5c-7805dfcf618c-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-9d24f\" (UID: \"bdc18c51-ef7d-45ee-9e5c-7805dfcf618c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f"
Sep 18 13:16:23.383104 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.383078    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bdc18c51-ef7d-45ee-9e5c-7805dfcf618c-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-9d24f\" (UID: \"bdc18c51-ef7d-45ee-9e5c-7805dfcf618c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f"
Sep 18 13:16:23.385135 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.385109    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bdc18c51-ef7d-45ee-9e5c-7805dfcf618c-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-9d24f\" (UID: \"bdc18c51-ef7d-45ee-9e5c-7805dfcf618c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f"
Sep 18 13:16:23.385841 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.385821    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bdc18c51-ef7d-45ee-9e5c-7805dfcf618c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-9d24f\" (UID: \"bdc18c51-ef7d-45ee-9e5c-7805dfcf618c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f"
Sep 18 13:16:23.391003 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.390956    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq4jm\" (UniqueName: \"kubernetes.io/projected/bdc18c51-ef7d-45ee-9e5c-7805dfcf618c-kube-api-access-nq4jm\") pod \"kube-state-metrics-84c885658c-9d24f\" (UID: \"bdc18c51-ef7d-45ee-9e5c-7805dfcf618c\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f"
Sep 18 13:16:23.483310 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.483223    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zw6m\" (UniqueName: \"kubernetes.io/projected/6c168813-ac20-48b4-b2a3-48664b28466a-kube-api-access-2zw6m\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.483310 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.483284    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6c168813-ac20-48b4-b2a3-48664b28466a-node-exporter-textfile\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.483537 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.483314    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6c168813-ac20-48b4-b2a3-48664b28466a-root\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.483537 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.483354    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6c168813-ac20-48b4-b2a3-48664b28466a-node-exporter-wtmp\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.483537 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.483381    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6c168813-ac20-48b4-b2a3-48664b28466a-node-exporter-tls\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.483537 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.483423    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6c168813-ac20-48b4-b2a3-48664b28466a-root\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.483537 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:16:23.483484    2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 18 13:16:23.483537 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.483501    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c168813-ac20-48b4-b2a3-48664b28466a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.483537 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.483524    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6c168813-ac20-48b4-b2a3-48664b28466a-node-exporter-wtmp\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.483537 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:16:23.483537    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c168813-ac20-48b4-b2a3-48664b28466a-node-exporter-tls podName:6c168813-ac20-48b4-b2a3-48664b28466a nodeName:}" failed. No retries permitted until 2026-09-18 13:16:23.983517892 +0000 UTC m=+92.418394295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6c168813-ac20-48b4-b2a3-48664b28466a-node-exporter-tls") pod "node-exporter-tf756" (UID: "6c168813-ac20-48b4-b2a3-48664b28466a") : secret "node-exporter-tls" not found
Sep 18 13:16:23.483981 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.483577    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c168813-ac20-48b4-b2a3-48664b28466a-sys\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.483981 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.483643    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6c168813-ac20-48b4-b2a3-48664b28466a-node-exporter-accelerators-collector-config\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.483981 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.483668    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c168813-ac20-48b4-b2a3-48664b28466a-metrics-client-ca\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.483981 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.483645    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c168813-ac20-48b4-b2a3-48664b28466a-sys\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.483981 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.483721    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6c168813-ac20-48b4-b2a3-48664b28466a-node-exporter-textfile\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.484202 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.484190    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c168813-ac20-48b4-b2a3-48664b28466a-metrics-client-ca\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.484257 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.484237    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6c168813-ac20-48b4-b2a3-48664b28466a-node-exporter-accelerators-collector-config\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.486288 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.486263    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c168813-ac20-48b4-b2a3-48664b28466a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.505513 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.505481    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zw6m\" (UniqueName: \"kubernetes.io/projected/6c168813-ac20-48b4-b2a3-48664b28466a-kube-api-access-2zw6m\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.508409 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.508386    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f"
Sep 18 13:16:23.988690 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.988651    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6c168813-ac20-48b4-b2a3-48664b28466a-node-exporter-tls\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:23.991274 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:23.991249    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6c168813-ac20-48b4-b2a3-48664b28466a-node-exporter-tls\") pod \"node-exporter-tf756\" (UID: \"6c168813-ac20-48b4-b2a3-48664b28466a\") " pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:24.117553 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.117520    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tf756"
Sep 18 13:16:24.289807 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.289703    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 13:16:24.294163 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.294137    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.298537 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.298509    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 13:16:24.298961 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.298939    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 13:16:24.299194 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.299174    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 13:16:24.299403 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.299383    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zkjlc\""
Sep 18 13:16:24.299587 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.299563    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 13:16:24.299697 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.299636    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 13:16:24.299764 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.299571    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 13:16:24.299988 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.299967    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 13:16:24.300068 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.299988    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 13:16:24.300068 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.299976    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 13:16:24.317875 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.317846    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 13:16:24.393248 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.393215    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.393442 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.393269    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.393442 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.393298    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-config-volume\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.393442 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.393317    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-web-config\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.393442 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.393344    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3da602f1-09f1-4d56-b2c2-659302e87a7d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.393442 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.393374    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3da602f1-09f1-4d56-b2c2-659302e87a7d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.393442 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.393395    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.393442 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.393417    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3da602f1-09f1-4d56-b2c2-659302e87a7d-config-out\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.393781 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.393512    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.393781 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.393562    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.393781 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.393661    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3da602f1-09f1-4d56-b2c2-659302e87a7d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.393781 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.393706    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3da602f1-09f1-4d56-b2c2-659302e87a7d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.393781 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.393748    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgk96\" (UniqueName: \"kubernetes.io/projected/3da602f1-09f1-4d56-b2c2-659302e87a7d-kube-api-access-mgk96\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.494871 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.494830    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.494871 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.494879    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.495126 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.494906    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3da602f1-09f1-4d56-b2c2-659302e87a7d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.495126 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.494931    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3da602f1-09f1-4d56-b2c2-659302e87a7d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.495126 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.494961    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgk96\" (UniqueName: \"kubernetes.io/projected/3da602f1-09f1-4d56-b2c2-659302e87a7d-kube-api-access-mgk96\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.495126 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:16:24.495007    2566 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 18 13:16:24.495126 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:16:24.495092    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-main-tls podName:3da602f1-09f1-4d56-b2c2-659302e87a7d nodeName:}" failed. No retries permitted until 2026-09-18 13:16:24.995069578 +0000 UTC m=+93.429945974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "3da602f1-09f1-4d56-b2c2-659302e87a7d") : secret "alertmanager-main-tls" not found
Sep 18 13:16:24.495423 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:16:24.495147    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3da602f1-09f1-4d56-b2c2-659302e87a7d-alertmanager-trusted-ca-bundle podName:3da602f1-09f1-4d56-b2c2-659302e87a7d nodeName:}" failed. No retries permitted until 2026-09-18 13:16:24.995130585 +0000 UTC m=+93.430006966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/3da602f1-09f1-4d56-b2c2-659302e87a7d-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "3da602f1-09f1-4d56-b2c2-659302e87a7d") : configmap references non-existent config key: ca-bundle.crt
Sep 18 13:16:24.495423 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.495011    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.495423 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.495211    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.495423 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.495246    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-config-volume\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.495423 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.495271    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-web-config\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.495423 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.495303    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3da602f1-09f1-4d56-b2c2-659302e87a7d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.495423 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.495339    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3da602f1-09f1-4d56-b2c2-659302e87a7d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.495423 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.495364    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.495423 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.495392    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3da602f1-09f1-4d56-b2c2-659302e87a7d-config-out\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.495886 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.495787    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3da602f1-09f1-4d56-b2c2-659302e87a7d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.498289 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.498244    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3da602f1-09f1-4d56-b2c2-659302e87a7d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.499039 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.499021    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3da602f1-09f1-4d56-b2c2-659302e87a7d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.499176 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.499148    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.499288 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.499225    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.499371 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.499348    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-web-config\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.499474 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.499450    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3da602f1-09f1-4d56-b2c2-659302e87a7d-config-out\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.500266 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.500243    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.500978 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.500941    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.501303 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.501280    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-config-volume\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:24.530869 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:24.530835    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgk96\" (UniqueName: \"kubernetes.io/projected/3da602f1-09f1-4d56-b2c2-659302e87a7d-kube-api-access-mgk96\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:25.001001 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:25.000959    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:25.001181 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:25.001019    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3da602f1-09f1-4d56-b2c2-659302e87a7d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:25.001917 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:25.001891    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3da602f1-09f1-4d56-b2c2-659302e87a7d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:25.003805 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:25.003780    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:25.206951 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:25.206909    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:16:28.611449 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:28.611303    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-vtcgz" event={"ID":"e167d2a4-7a91-46b1-a5e3-da0a27ede351","Type":"ContainerStarted","Data":"4e6a25c648d3d1707ba31aa1b084972eb7822ce1717a6ecbb25ff95c2d27a232"}
Sep 18 13:16:28.611981 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:28.611516    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-vtcgz"
Sep 18 13:16:28.612714 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:28.612690    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tf756" event={"ID":"6c168813-ac20-48b4-b2a3-48664b28466a","Type":"ContainerStarted","Data":"96667c21043e2712997943f8c62102dcb976e00310a089b0f357925982653e1b"}
Sep 18 13:16:28.613340 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:28.613312    2566 patch_prober.go:28] interesting pod/downloads-6594f4c75-vtcgz container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.18:8080/\": dial tcp 10.133.0.18:8080: connect: connection refused" start-of-body=
Sep 18 13:16:28.613410 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:28.613363    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6594f4c75-vtcgz" podUID="e167d2a4-7a91-46b1-a5e3-da0a27ede351" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.18:8080/\": dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 18 13:16:28.630837 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:28.630812    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-9d24f"]
Sep 18 13:16:28.638032 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:28.637799    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-vtcgz" podStartSLOduration=1.849259268 podStartE2EDuration="18.637780731s" totalImagesPullingTime="16.788521463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:16:28.632484479 +0000 UTC m=+97.067360883" watchObservedRunningTime="2026-09-18 13:16:28.637780731 +0000 UTC m=+97.072657136"
Sep 18 13:16:28.642902 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:28.639866    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 13:16:28.644739 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:16:28.644714    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da602f1_09f1_4d56_b2c2_659302e87a7d.slice/crio-73afedf5398234aca50154407c3c79a5b66ac59e98b2c5062bb8e58ed53fb523 WatchSource:0}: Error finding container 73afedf5398234aca50154407c3c79a5b66ac59e98b2c5062bb8e58ed53fb523: Status 404 returned error can't find the container with id 73afedf5398234aca50154407c3c79a5b66ac59e98b2c5062bb8e58ed53fb523
Sep 18 13:16:29.623354 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:29.623322    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3da602f1-09f1-4d56-b2c2-659302e87a7d","Type":"ContainerStarted","Data":"73afedf5398234aca50154407c3c79a5b66ac59e98b2c5062bb8e58ed53fb523"}
Sep 18 13:16:29.625749 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:29.625667    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f" event={"ID":"bdc18c51-ef7d-45ee-9e5c-7805dfcf618c","Type":"ContainerStarted","Data":"53199ec0c45c2000bd5483285bd035f3518ca022af4259b79cffccba93f135c8"}
Sep 18 13:16:29.644978 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:29.644884    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-vtcgz"
Sep 18 13:16:30.632353 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:30.632309    2566 generic.go:356] "Generic (PLEG): container finished" podID="6c168813-ac20-48b4-b2a3-48664b28466a" containerID="12dcd806edf20ef4db1583d19b52bb4561661b52f0d4987c6f634fb9906bf5d6" exitCode=0
Sep 18 13:16:30.632859 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:30.632417    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tf756" event={"ID":"6c168813-ac20-48b4-b2a3-48664b28466a","Type":"ContainerDied","Data":"12dcd806edf20ef4db1583d19b52bb4561661b52f0d4987c6f634fb9906bf5d6"}
Sep 18 13:16:31.477136 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:31.477050    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vdj8k"
Sep 18 13:16:31.650481 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:31.650427    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tf756" event={"ID":"6c168813-ac20-48b4-b2a3-48664b28466a","Type":"ContainerStarted","Data":"bae4239b2b6834a0f7e47fbf222955bc15b4000c97b047aed8502536383450a7"}
Sep 18 13:16:31.650984 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:31.650563    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tf756" event={"ID":"6c168813-ac20-48b4-b2a3-48664b28466a","Type":"ContainerStarted","Data":"a01d97744ee129b640f4f667fa43dc89b992d8de7e5f7802321e0c35b17ac1ef"}
Sep 18 13:16:31.653405 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:31.653371    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f" event={"ID":"bdc18c51-ef7d-45ee-9e5c-7805dfcf618c","Type":"ContainerStarted","Data":"25fe6fa448f689c38b71ed394b77bb8e2fd8d3126f0d3cbbc5c97ba60fd8dcd7"}
Sep 18 13:16:31.653547 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:31.653435    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f" event={"ID":"bdc18c51-ef7d-45ee-9e5c-7805dfcf618c","Type":"ContainerStarted","Data":"cf752fbc8110d5c3577cd678612e17fa607f076685c00d4d63521ee04bd386b0"}
Sep 18 13:16:31.653547 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:31.653492    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f" event={"ID":"bdc18c51-ef7d-45ee-9e5c-7805dfcf618c","Type":"ContainerStarted","Data":"084cd81ada517af056850ab097d0a90117d3cffa595f29f9bed7d13fe0534cbe"}
Sep 18 13:16:31.655565 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:31.655538    2566 generic.go:356] "Generic (PLEG): container finished" podID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerID="5cc98f8276415ff6219c7249dc43ac60e8c0e02385b9396ef777d4946832c5b9" exitCode=0
Sep 18 13:16:31.655698 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:31.655618    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3da602f1-09f1-4d56-b2c2-659302e87a7d","Type":"ContainerDied","Data":"5cc98f8276415ff6219c7249dc43ac60e8c0e02385b9396ef777d4946832c5b9"}
Sep 18 13:16:31.669214 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:31.669158    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tf756" podStartSLOduration=7.363093506 podStartE2EDuration="8.66913887s" totalImagesPullingTime="1.207490332s" totalInitContainerRuntime="98.555032ms" isStatefulPod=true podCreationTimestamp="2026-09-18 13:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:16:31.667361867 +0000 UTC m=+100.102238272" watchObservedRunningTime="2026-09-18 13:16:31.66913887 +0000 UTC m=+100.104015274"
Sep 18 13:16:31.710221 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:31.710153    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-9d24f" podStartSLOduration=6.439870315 podStartE2EDuration="8.710122729s" totalImagesPullingTime="2.270252414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:16:31.708077292 +0000 UTC m=+100.142953710" watchObservedRunningTime="2026-09-18 13:16:31.710122729 +0000 UTC m=+100.144999135"
Sep 18 13:16:34.676445 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:34.676357    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3da602f1-09f1-4d56-b2c2-659302e87a7d","Type":"ContainerStarted","Data":"bc1213bcc78a51a0c7b63c387f4a427543cc95bcb798a35b272bbc38353826fa"}
Sep 18 13:16:34.676915 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:34.676444    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3da602f1-09f1-4d56-b2c2-659302e87a7d","Type":"ContainerStarted","Data":"c224ebb8bae81ab1817e0d8bb7efc4b7aacb813285db7cceca8631763fa12e6c"}
Sep 18 13:16:34.676915 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:34.676484    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3da602f1-09f1-4d56-b2c2-659302e87a7d","Type":"ContainerStarted","Data":"ca93863d0dceebff9b92afea95b091bfc44bda5a02041b5e9c264f8dfdf03d2b"}
Sep 18 13:16:34.676915 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:34.676524    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3da602f1-09f1-4d56-b2c2-659302e87a7d","Type":"ContainerStarted","Data":"c0552b45456c2e51f6382d761d017462fbfed5abc9d8eddcd9127f521cbdef59"}
Sep 18 13:16:34.676915 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:34.676563    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3da602f1-09f1-4d56-b2c2-659302e87a7d","Type":"ContainerStarted","Data":"40b3526a52867bfe69c5ce75220096169b67092e28f19dd720284a577fc348ef"}
Sep 18 13:16:36.111044 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.110954    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-668879999c-dfvz7" podUID="0bb07b38-9199-424d-9fb7-3ffa57def30a" containerName="registry" containerID="cri-o://9a4737325d61cfe2f696084433aab8569b9837522063c68f0433176990426736" gracePeriod=30
Sep 18 13:16:36.407873 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.407836    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:16:36.509502 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.509454    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0bb07b38-9199-424d-9fb7-3ffa57def30a-installation-pull-secrets\") pod \"0bb07b38-9199-424d-9fb7-3ffa57def30a\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") "
Sep 18 13:16:36.509502 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.509499    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-certificates\") pod \"0bb07b38-9199-424d-9fb7-3ffa57def30a\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") "
Sep 18 13:16:36.509785 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.509542    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0bb07b38-9199-424d-9fb7-3ffa57def30a-trusted-ca\") pod \"0bb07b38-9199-424d-9fb7-3ffa57def30a\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") "
Sep 18 13:16:36.509785 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.509568    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnmrq\" (UniqueName: \"kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-kube-api-access-fnmrq\") pod \"0bb07b38-9199-424d-9fb7-3ffa57def30a\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") "
Sep 18 13:16:36.509785 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.509610    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0bb07b38-9199-424d-9fb7-3ffa57def30a-image-registry-private-configuration\") pod \"0bb07b38-9199-424d-9fb7-3ffa57def30a\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") "
Sep 18 13:16:36.509785 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.509655    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0bb07b38-9199-424d-9fb7-3ffa57def30a-ca-trust-extracted\") pod \"0bb07b38-9199-424d-9fb7-3ffa57def30a\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") "
Sep 18 13:16:36.509785 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.509723    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-tls\") pod \"0bb07b38-9199-424d-9fb7-3ffa57def30a\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") "
Sep 18 13:16:36.509785 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.509753    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-bound-sa-token\") pod \"0bb07b38-9199-424d-9fb7-3ffa57def30a\" (UID: \"0bb07b38-9199-424d-9fb7-3ffa57def30a\") "
Sep 18 13:16:36.510070 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.509975    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "0bb07b38-9199-424d-9fb7-3ffa57def30a" (UID: "0bb07b38-9199-424d-9fb7-3ffa57def30a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:16:36.510070 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.510029    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb07b38-9199-424d-9fb7-3ffa57def30a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "0bb07b38-9199-424d-9fb7-3ffa57def30a" (UID: "0bb07b38-9199-424d-9fb7-3ffa57def30a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:16:36.513037 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.512965    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-kube-api-access-fnmrq" (OuterVolumeSpecName: "kube-api-access-fnmrq") pod "0bb07b38-9199-424d-9fb7-3ffa57def30a" (UID: "0bb07b38-9199-424d-9fb7-3ffa57def30a"). InnerVolumeSpecName "kube-api-access-fnmrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:16:36.513167 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.513096    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bb07b38-9199-424d-9fb7-3ffa57def30a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "0bb07b38-9199-424d-9fb7-3ffa57def30a" (UID: "0bb07b38-9199-424d-9fb7-3ffa57def30a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:16:36.513167 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.513104    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "0bb07b38-9199-424d-9fb7-3ffa57def30a" (UID: "0bb07b38-9199-424d-9fb7-3ffa57def30a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:16:36.513285 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.513176    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0bb07b38-9199-424d-9fb7-3ffa57def30a" (UID: "0bb07b38-9199-424d-9fb7-3ffa57def30a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:16:36.513285 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.513227    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bb07b38-9199-424d-9fb7-3ffa57def30a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "0bb07b38-9199-424d-9fb7-3ffa57def30a" (UID: "0bb07b38-9199-424d-9fb7-3ffa57def30a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:16:36.522798 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.522763    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb07b38-9199-424d-9fb7-3ffa57def30a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "0bb07b38-9199-424d-9fb7-3ffa57def30a" (UID: "0bb07b38-9199-424d-9fb7-3ffa57def30a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:16:36.610853 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.610816    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-tls\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:16:36.610853 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.610849    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-bound-sa-token\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:16:36.610853 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.610858    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0bb07b38-9199-424d-9fb7-3ffa57def30a-installation-pull-secrets\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:16:36.611096 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.610868    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0bb07b38-9199-424d-9fb7-3ffa57def30a-registry-certificates\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:16:36.611096 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.610878    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0bb07b38-9199-424d-9fb7-3ffa57def30a-trusted-ca\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:16:36.611096 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.610891    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnmrq\" (UniqueName: \"kubernetes.io/projected/0bb07b38-9199-424d-9fb7-3ffa57def30a-kube-api-access-fnmrq\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:16:36.611096 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.610906    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0bb07b38-9199-424d-9fb7-3ffa57def30a-image-registry-private-configuration\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:16:36.611096 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.610921    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0bb07b38-9199-424d-9fb7-3ffa57def30a-ca-trust-extracted\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:16:36.687746 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.687662    2566 generic.go:356] "Generic (PLEG): container finished" podID="0bb07b38-9199-424d-9fb7-3ffa57def30a" containerID="9a4737325d61cfe2f696084433aab8569b9837522063c68f0433176990426736" exitCode=0
Sep 18 13:16:36.687746 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.687703    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-668879999c-dfvz7" event={"ID":"0bb07b38-9199-424d-9fb7-3ffa57def30a","Type":"ContainerDied","Data":"9a4737325d61cfe2f696084433aab8569b9837522063c68f0433176990426736"}
Sep 18 13:16:36.687975 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.687772    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-668879999c-dfvz7" event={"ID":"0bb07b38-9199-424d-9fb7-3ffa57def30a","Type":"ContainerDied","Data":"1c3df0b02a0878264582aab2a5be2b04e2fbe7b4fda906ad6f163958dac76c4e"}
Sep 18 13:16:36.687975 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.687725    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-668879999c-dfvz7"
Sep 18 13:16:36.687975 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.687801    2566 scope.go:117] "RemoveContainer" containerID="9a4737325d61cfe2f696084433aab8569b9837522063c68f0433176990426736"
Sep 18 13:16:36.692514 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.692486    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3da602f1-09f1-4d56-b2c2-659302e87a7d","Type":"ContainerStarted","Data":"1ed3c2bc8edf1be246cb2731c0f8728757b802c954bf3c79a60ac9b7d4100ad1"}
Sep 18 13:16:36.698999 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.698700    2566 scope.go:117] "RemoveContainer" containerID="9a4737325d61cfe2f696084433aab8569b9837522063c68f0433176990426736"
Sep 18 13:16:36.699298 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:16:36.699267    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4737325d61cfe2f696084433aab8569b9837522063c68f0433176990426736\": container with ID starting with 9a4737325d61cfe2f696084433aab8569b9837522063c68f0433176990426736 not found: ID does not exist" containerID="9a4737325d61cfe2f696084433aab8569b9837522063c68f0433176990426736"
Sep 18 13:16:36.699405 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.699307    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4737325d61cfe2f696084433aab8569b9837522063c68f0433176990426736"} err="failed to get container status \"9a4737325d61cfe2f696084433aab8569b9837522063c68f0433176990426736\": rpc error: code = NotFound desc = could not find container \"9a4737325d61cfe2f696084433aab8569b9837522063c68f0433176990426736\": container with ID starting with 9a4737325d61cfe2f696084433aab8569b9837522063c68f0433176990426736 not found: ID does not exist"
Sep 18 13:16:36.725109 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.725053    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=6.929711226 podStartE2EDuration="12.725030889s" totalImagesPullingTime="5.757575422s" totalInitContainerRuntime="37.744241ms" isStatefulPod=false podCreationTimestamp="2026-09-18 13:16:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:16:36.72340531 +0000 UTC m=+105.158281715" watchObservedRunningTime="2026-09-18 13:16:36.725030889 +0000 UTC m=+105.159907296"
Sep 18 13:16:36.741563 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.741528    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-668879999c-dfvz7"]
Sep 18 13:16:36.747561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:36.747217    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-668879999c-dfvz7"]
Sep 18 13:16:38.076204 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:38.076168    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bb07b38-9199-424d-9fb7-3ffa57def30a" path="/var/lib/kubelet/pods/0bb07b38-9199-424d-9fb7-3ffa57def30a/volumes"
Sep 18 13:16:50.751203 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:50.751167    2566 generic.go:356] "Generic (PLEG): container finished" podID="1555f97d-9829-4708-b2c7-b2afb963ad05" containerID="cb5e0ffc181b64762c7c732b900b0743c8aa6b6e8e67e5dff33590ad4b861dce" exitCode=0
Sep 18 13:16:50.751693 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:50.751231    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-c9tj5" event={"ID":"1555f97d-9829-4708-b2c7-b2afb963ad05","Type":"ContainerDied","Data":"cb5e0ffc181b64762c7c732b900b0743c8aa6b6e8e67e5dff33590ad4b861dce"}
Sep 18 13:16:50.751693 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:50.751529    2566 scope.go:117] "RemoveContainer" containerID="cb5e0ffc181b64762c7c732b900b0743c8aa6b6e8e67e5dff33590ad4b861dce"
Sep 18 13:16:51.757287 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:51.757253    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-c9tj5" event={"ID":"1555f97d-9829-4708-b2c7-b2afb963ad05","Type":"ContainerStarted","Data":"2bb843f085001a556f3d07117386143f4f44b74e1b0ae9e8ea51664a46f5eda2"}
Sep 18 13:16:55.772629 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:55.772573    2566 generic.go:356] "Generic (PLEG): container finished" podID="c7cd87f4-e7a0-431e-9cd9-73fbe25e3a7f" containerID="3005a5463449ca9ef6b20740e29be5f9fdd0366131eaa23623fc440eb9f82ad7" exitCode=0
Sep 18 13:16:55.773118 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:55.772645    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-58qsl" event={"ID":"c7cd87f4-e7a0-431e-9cd9-73fbe25e3a7f","Type":"ContainerDied","Data":"3005a5463449ca9ef6b20740e29be5f9fdd0366131eaa23623fc440eb9f82ad7"}
Sep 18 13:16:55.773118 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:55.772936    2566 scope.go:117] "RemoveContainer" containerID="3005a5463449ca9ef6b20740e29be5f9fdd0366131eaa23623fc440eb9f82ad7"
Sep 18 13:16:56.778822 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:56.778785    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-58qsl" event={"ID":"c7cd87f4-e7a0-431e-9cd9-73fbe25e3a7f","Type":"ContainerStarted","Data":"892a66c2a05e521a584cd79c42f5995ab9dbd29de7b84faeb162cdfad785c32a"}
Sep 18 13:16:57.940800 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:16:57.940765    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cn6sm_2452828a-bce2-4da3-a477-88d9d048a4b3/serve-healthcheck-canary/0.log"
Sep 18 13:17:43.638495 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:43.638452    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 13:17:43.639095 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:43.638871    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="alertmanager" containerID="cri-o://40b3526a52867bfe69c5ce75220096169b67092e28f19dd720284a577fc348ef" gracePeriod=120
Sep 18 13:17:43.639095 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:43.638904    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="kube-rbac-proxy-metric" containerID="cri-o://bc1213bcc78a51a0c7b63c387f4a427543cc95bcb798a35b272bbc38353826fa" gracePeriod=120
Sep 18 13:17:43.639095 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:43.638949    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="prom-label-proxy" containerID="cri-o://1ed3c2bc8edf1be246cb2731c0f8728757b802c954bf3c79a60ac9b7d4100ad1" gracePeriod=120
Sep 18 13:17:43.639095 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:43.638987    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="config-reloader" containerID="cri-o://c0552b45456c2e51f6382d761d017462fbfed5abc9d8eddcd9127f521cbdef59" gracePeriod=120
Sep 18 13:17:43.639095 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:43.638971    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="kube-rbac-proxy-web" containerID="cri-o://ca93863d0dceebff9b92afea95b091bfc44bda5a02041b5e9c264f8dfdf03d2b" gracePeriod=120
Sep 18 13:17:43.639095 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:43.638995    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="kube-rbac-proxy" containerID="cri-o://c224ebb8bae81ab1817e0d8bb7efc4b7aacb813285db7cceca8631763fa12e6c" gracePeriod=120
Sep 18 13:17:43.957104 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:43.957001    2566 generic.go:356] "Generic (PLEG): container finished" podID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerID="1ed3c2bc8edf1be246cb2731c0f8728757b802c954bf3c79a60ac9b7d4100ad1" exitCode=0
Sep 18 13:17:43.957104 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:43.957032    2566 generic.go:356] "Generic (PLEG): container finished" podID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerID="bc1213bcc78a51a0c7b63c387f4a427543cc95bcb798a35b272bbc38353826fa" exitCode=0
Sep 18 13:17:43.957104 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:43.957042    2566 generic.go:356] "Generic (PLEG): container finished" podID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerID="c224ebb8bae81ab1817e0d8bb7efc4b7aacb813285db7cceca8631763fa12e6c" exitCode=0
Sep 18 13:17:43.957104 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:43.957049    2566 generic.go:356] "Generic (PLEG): container finished" podID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerID="c0552b45456c2e51f6382d761d017462fbfed5abc9d8eddcd9127f521cbdef59" exitCode=0
Sep 18 13:17:43.957104 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:43.957056    2566 generic.go:356] "Generic (PLEG): container finished" podID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerID="40b3526a52867bfe69c5ce75220096169b67092e28f19dd720284a577fc348ef" exitCode=0
Sep 18 13:17:43.957104 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:43.957070    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3da602f1-09f1-4d56-b2c2-659302e87a7d","Type":"ContainerDied","Data":"1ed3c2bc8edf1be246cb2731c0f8728757b802c954bf3c79a60ac9b7d4100ad1"}
Sep 18 13:17:43.957396 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:43.957136    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3da602f1-09f1-4d56-b2c2-659302e87a7d","Type":"ContainerDied","Data":"bc1213bcc78a51a0c7b63c387f4a427543cc95bcb798a35b272bbc38353826fa"}
Sep 18 13:17:43.957396 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:43.957162    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3da602f1-09f1-4d56-b2c2-659302e87a7d","Type":"ContainerDied","Data":"c224ebb8bae81ab1817e0d8bb7efc4b7aacb813285db7cceca8631763fa12e6c"}
Sep 18 13:17:43.957396 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:43.957186    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3da602f1-09f1-4d56-b2c2-659302e87a7d","Type":"ContainerDied","Data":"c0552b45456c2e51f6382d761d017462fbfed5abc9d8eddcd9127f521cbdef59"}
Sep 18 13:17:43.957396 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:43.957213    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3da602f1-09f1-4d56-b2c2-659302e87a7d","Type":"ContainerDied","Data":"40b3526a52867bfe69c5ce75220096169b67092e28f19dd720284a577fc348ef"}
Sep 18 13:17:44.883181 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.883154    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:44.945554 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.945521    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-kube-rbac-proxy\") pod \"3da602f1-09f1-4d56-b2c2-659302e87a7d\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") "
Sep 18 13:17:44.945752 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.945562    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"3da602f1-09f1-4d56-b2c2-659302e87a7d\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") "
Sep 18 13:17:44.945752 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.945603    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-kube-rbac-proxy-web\") pod \"3da602f1-09f1-4d56-b2c2-659302e87a7d\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") "
Sep 18 13:17:44.945752 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.945632    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3da602f1-09f1-4d56-b2c2-659302e87a7d-config-out\") pod \"3da602f1-09f1-4d56-b2c2-659302e87a7d\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") "
Sep 18 13:17:44.945752 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.945690    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-cluster-tls-config\") pod \"3da602f1-09f1-4d56-b2c2-659302e87a7d\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") "
Sep 18 13:17:44.945752 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.945727    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3da602f1-09f1-4d56-b2c2-659302e87a7d-alertmanager-main-db\") pod \"3da602f1-09f1-4d56-b2c2-659302e87a7d\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") "
Sep 18 13:17:44.945752 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.945754    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-main-tls\") pod \"3da602f1-09f1-4d56-b2c2-659302e87a7d\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") "
Sep 18 13:17:44.946010 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.945788    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-web-config\") pod \"3da602f1-09f1-4d56-b2c2-659302e87a7d\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") "
Sep 18 13:17:44.946010 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.945812    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3da602f1-09f1-4d56-b2c2-659302e87a7d-metrics-client-ca\") pod \"3da602f1-09f1-4d56-b2c2-659302e87a7d\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") "
Sep 18 13:17:44.946010 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.945836    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3da602f1-09f1-4d56-b2c2-659302e87a7d-tls-assets\") pod \"3da602f1-09f1-4d56-b2c2-659302e87a7d\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") "
Sep 18 13:17:44.946010 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.945866    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgk96\" (UniqueName: \"kubernetes.io/projected/3da602f1-09f1-4d56-b2c2-659302e87a7d-kube-api-access-mgk96\") pod \"3da602f1-09f1-4d56-b2c2-659302e87a7d\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") "
Sep 18 13:17:44.946010 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.945897    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3da602f1-09f1-4d56-b2c2-659302e87a7d-alertmanager-trusted-ca-bundle\") pod \"3da602f1-09f1-4d56-b2c2-659302e87a7d\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") "
Sep 18 13:17:44.946229 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.946116    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-config-volume\") pod \"3da602f1-09f1-4d56-b2c2-659302e87a7d\" (UID: \"3da602f1-09f1-4d56-b2c2-659302e87a7d\") "
Sep 18 13:17:44.946970 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.946933    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3da602f1-09f1-4d56-b2c2-659302e87a7d-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "3da602f1-09f1-4d56-b2c2-659302e87a7d" (UID: "3da602f1-09f1-4d56-b2c2-659302e87a7d"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:17:44.947452 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.947424    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da602f1-09f1-4d56-b2c2-659302e87a7d-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "3da602f1-09f1-4d56-b2c2-659302e87a7d" (UID: "3da602f1-09f1-4d56-b2c2-659302e87a7d"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:17:44.947558 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.947523    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3da602f1-09f1-4d56-b2c2-659302e87a7d-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "3da602f1-09f1-4d56-b2c2-659302e87a7d" (UID: "3da602f1-09f1-4d56-b2c2-659302e87a7d"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:17:44.950102 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.950038    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "3da602f1-09f1-4d56-b2c2-659302e87a7d" (UID: "3da602f1-09f1-4d56-b2c2-659302e87a7d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:17:44.950339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.950299    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da602f1-09f1-4d56-b2c2-659302e87a7d-kube-api-access-mgk96" (OuterVolumeSpecName: "kube-api-access-mgk96") pod "3da602f1-09f1-4d56-b2c2-659302e87a7d" (UID: "3da602f1-09f1-4d56-b2c2-659302e87a7d"). InnerVolumeSpecName "kube-api-access-mgk96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:17:44.950835 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.950803    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "3da602f1-09f1-4d56-b2c2-659302e87a7d" (UID: "3da602f1-09f1-4d56-b2c2-659302e87a7d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:17:44.951294 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.951255    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da602f1-09f1-4d56-b2c2-659302e87a7d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3da602f1-09f1-4d56-b2c2-659302e87a7d" (UID: "3da602f1-09f1-4d56-b2c2-659302e87a7d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:17:44.951691 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.951662    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da602f1-09f1-4d56-b2c2-659302e87a7d-config-out" (OuterVolumeSpecName: "config-out") pod "3da602f1-09f1-4d56-b2c2-659302e87a7d" (UID: "3da602f1-09f1-4d56-b2c2-659302e87a7d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:17:44.952173 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.952134    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "3da602f1-09f1-4d56-b2c2-659302e87a7d" (UID: "3da602f1-09f1-4d56-b2c2-659302e87a7d"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:17:44.952382 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.952347    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "3da602f1-09f1-4d56-b2c2-659302e87a7d" (UID: "3da602f1-09f1-4d56-b2c2-659302e87a7d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:17:44.952545 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.952517    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-config-volume" (OuterVolumeSpecName: "config-volume") pod "3da602f1-09f1-4d56-b2c2-659302e87a7d" (UID: "3da602f1-09f1-4d56-b2c2-659302e87a7d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:17:44.954095 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.954068    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "3da602f1-09f1-4d56-b2c2-659302e87a7d" (UID: "3da602f1-09f1-4d56-b2c2-659302e87a7d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:17:44.966522 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.966406    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-web-config" (OuterVolumeSpecName: "web-config") pod "3da602f1-09f1-4d56-b2c2-659302e87a7d" (UID: "3da602f1-09f1-4d56-b2c2-659302e87a7d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:17:44.969981 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.969949    2566 generic.go:356] "Generic (PLEG): container finished" podID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerID="ca93863d0dceebff9b92afea95b091bfc44bda5a02041b5e9c264f8dfdf03d2b" exitCode=0
Sep 18 13:17:44.970105 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.970030    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:44.970105 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.970028    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3da602f1-09f1-4d56-b2c2-659302e87a7d","Type":"ContainerDied","Data":"ca93863d0dceebff9b92afea95b091bfc44bda5a02041b5e9c264f8dfdf03d2b"}
Sep 18 13:17:44.970216 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.970149    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3da602f1-09f1-4d56-b2c2-659302e87a7d","Type":"ContainerDied","Data":"73afedf5398234aca50154407c3c79a5b66ac59e98b2c5062bb8e58ed53fb523"}
Sep 18 13:17:44.970216 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.970204    2566 scope.go:117] "RemoveContainer" containerID="1ed3c2bc8edf1be246cb2731c0f8728757b802c954bf3c79a60ac9b7d4100ad1"
Sep 18 13:17:44.978447 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.978414    2566 scope.go:117] "RemoveContainer" containerID="bc1213bcc78a51a0c7b63c387f4a427543cc95bcb798a35b272bbc38353826fa"
Sep 18 13:17:44.986535 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.986515    2566 scope.go:117] "RemoveContainer" containerID="c224ebb8bae81ab1817e0d8bb7efc4b7aacb813285db7cceca8631763fa12e6c"
Sep 18 13:17:44.996281 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:44.996255    2566 scope.go:117] "RemoveContainer" containerID="ca93863d0dceebff9b92afea95b091bfc44bda5a02041b5e9c264f8dfdf03d2b"
Sep 18 13:17:45.000106 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.000076    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 13:17:45.004166 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.004135    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 13:17:45.005033 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.005016    2566 scope.go:117] "RemoveContainer" containerID="c0552b45456c2e51f6382d761d017462fbfed5abc9d8eddcd9127f521cbdef59"
Sep 18 13:17:45.012687 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.012666    2566 scope.go:117] "RemoveContainer" containerID="40b3526a52867bfe69c5ce75220096169b67092e28f19dd720284a577fc348ef"
Sep 18 13:17:45.020011 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.019994    2566 scope.go:117] "RemoveContainer" containerID="5cc98f8276415ff6219c7249dc43ac60e8c0e02385b9396ef777d4946832c5b9"
Sep 18 13:17:45.027076 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.027055    2566 scope.go:117] "RemoveContainer" containerID="1ed3c2bc8edf1be246cb2731c0f8728757b802c954bf3c79a60ac9b7d4100ad1"
Sep 18 13:17:45.027366 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:17:45.027339    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ed3c2bc8edf1be246cb2731c0f8728757b802c954bf3c79a60ac9b7d4100ad1\": container with ID starting with 1ed3c2bc8edf1be246cb2731c0f8728757b802c954bf3c79a60ac9b7d4100ad1 not found: ID does not exist" containerID="1ed3c2bc8edf1be246cb2731c0f8728757b802c954bf3c79a60ac9b7d4100ad1"
Sep 18 13:17:45.027505 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.027374    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ed3c2bc8edf1be246cb2731c0f8728757b802c954bf3c79a60ac9b7d4100ad1"} err="failed to get container status \"1ed3c2bc8edf1be246cb2731c0f8728757b802c954bf3c79a60ac9b7d4100ad1\": rpc error: code = NotFound desc = could not find container \"1ed3c2bc8edf1be246cb2731c0f8728757b802c954bf3c79a60ac9b7d4100ad1\": container with ID starting with 1ed3c2bc8edf1be246cb2731c0f8728757b802c954bf3c79a60ac9b7d4100ad1 not found: ID does not exist"
Sep 18 13:17:45.027505 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.027394    2566 scope.go:117] "RemoveContainer" containerID="bc1213bcc78a51a0c7b63c387f4a427543cc95bcb798a35b272bbc38353826fa"
Sep 18 13:17:45.027664 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:17:45.027643    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc1213bcc78a51a0c7b63c387f4a427543cc95bcb798a35b272bbc38353826fa\": container with ID starting with bc1213bcc78a51a0c7b63c387f4a427543cc95bcb798a35b272bbc38353826fa not found: ID does not exist" containerID="bc1213bcc78a51a0c7b63c387f4a427543cc95bcb798a35b272bbc38353826fa"
Sep 18 13:17:45.027734 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.027674    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc1213bcc78a51a0c7b63c387f4a427543cc95bcb798a35b272bbc38353826fa"} err="failed to get container status \"bc1213bcc78a51a0c7b63c387f4a427543cc95bcb798a35b272bbc38353826fa\": rpc error: code = NotFound desc = could not find container \"bc1213bcc78a51a0c7b63c387f4a427543cc95bcb798a35b272bbc38353826fa\": container with ID starting with bc1213bcc78a51a0c7b63c387f4a427543cc95bcb798a35b272bbc38353826fa not found: ID does not exist"
Sep 18 13:17:45.027734 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.027696    2566 scope.go:117] "RemoveContainer" containerID="c224ebb8bae81ab1817e0d8bb7efc4b7aacb813285db7cceca8631763fa12e6c"
Sep 18 13:17:45.027944 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:17:45.027928    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c224ebb8bae81ab1817e0d8bb7efc4b7aacb813285db7cceca8631763fa12e6c\": container with ID starting with c224ebb8bae81ab1817e0d8bb7efc4b7aacb813285db7cceca8631763fa12e6c not found: ID does not exist" containerID="c224ebb8bae81ab1817e0d8bb7efc4b7aacb813285db7cceca8631763fa12e6c"
Sep 18 13:17:45.027992 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.027948    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c224ebb8bae81ab1817e0d8bb7efc4b7aacb813285db7cceca8631763fa12e6c"} err="failed to get container status \"c224ebb8bae81ab1817e0d8bb7efc4b7aacb813285db7cceca8631763fa12e6c\": rpc error: code = NotFound desc = could not find container \"c224ebb8bae81ab1817e0d8bb7efc4b7aacb813285db7cceca8631763fa12e6c\": container with ID starting with c224ebb8bae81ab1817e0d8bb7efc4b7aacb813285db7cceca8631763fa12e6c not found: ID does not exist"
Sep 18 13:17:45.027992 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.027962    2566 scope.go:117] "RemoveContainer" containerID="ca93863d0dceebff9b92afea95b091bfc44bda5a02041b5e9c264f8dfdf03d2b"
Sep 18 13:17:45.028174 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:17:45.028160    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca93863d0dceebff9b92afea95b091bfc44bda5a02041b5e9c264f8dfdf03d2b\": container with ID starting with ca93863d0dceebff9b92afea95b091bfc44bda5a02041b5e9c264f8dfdf03d2b not found: ID does not exist" containerID="ca93863d0dceebff9b92afea95b091bfc44bda5a02041b5e9c264f8dfdf03d2b"
Sep 18 13:17:45.028218 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.028176    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca93863d0dceebff9b92afea95b091bfc44bda5a02041b5e9c264f8dfdf03d2b"} err="failed to get container status \"ca93863d0dceebff9b92afea95b091bfc44bda5a02041b5e9c264f8dfdf03d2b\": rpc error: code = NotFound desc = could not find container \"ca93863d0dceebff9b92afea95b091bfc44bda5a02041b5e9c264f8dfdf03d2b\": container with ID starting with ca93863d0dceebff9b92afea95b091bfc44bda5a02041b5e9c264f8dfdf03d2b not found: ID does not exist"
Sep 18 13:17:45.028218 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.028188    2566 scope.go:117] "RemoveContainer" containerID="c0552b45456c2e51f6382d761d017462fbfed5abc9d8eddcd9127f521cbdef59"
Sep 18 13:17:45.028399 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:17:45.028382    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0552b45456c2e51f6382d761d017462fbfed5abc9d8eddcd9127f521cbdef59\": container with ID starting with c0552b45456c2e51f6382d761d017462fbfed5abc9d8eddcd9127f521cbdef59 not found: ID does not exist" containerID="c0552b45456c2e51f6382d761d017462fbfed5abc9d8eddcd9127f521cbdef59"
Sep 18 13:17:45.028438 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.028402    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0552b45456c2e51f6382d761d017462fbfed5abc9d8eddcd9127f521cbdef59"} err="failed to get container status \"c0552b45456c2e51f6382d761d017462fbfed5abc9d8eddcd9127f521cbdef59\": rpc error: code = NotFound desc = could not find container \"c0552b45456c2e51f6382d761d017462fbfed5abc9d8eddcd9127f521cbdef59\": container with ID starting with c0552b45456c2e51f6382d761d017462fbfed5abc9d8eddcd9127f521cbdef59 not found: ID does not exist"
Sep 18 13:17:45.028438 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.028414    2566 scope.go:117] "RemoveContainer" containerID="40b3526a52867bfe69c5ce75220096169b67092e28f19dd720284a577fc348ef"
Sep 18 13:17:45.028622 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:17:45.028585    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40b3526a52867bfe69c5ce75220096169b67092e28f19dd720284a577fc348ef\": container with ID starting with 40b3526a52867bfe69c5ce75220096169b67092e28f19dd720284a577fc348ef not found: ID does not exist" containerID="40b3526a52867bfe69c5ce75220096169b67092e28f19dd720284a577fc348ef"
Sep 18 13:17:45.028672 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.028626    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40b3526a52867bfe69c5ce75220096169b67092e28f19dd720284a577fc348ef"} err="failed to get container status \"40b3526a52867bfe69c5ce75220096169b67092e28f19dd720284a577fc348ef\": rpc error: code = NotFound desc = could not find container \"40b3526a52867bfe69c5ce75220096169b67092e28f19dd720284a577fc348ef\": container with ID starting with 40b3526a52867bfe69c5ce75220096169b67092e28f19dd720284a577fc348ef not found: ID does not exist"
Sep 18 13:17:45.028672 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.028637    2566 scope.go:117] "RemoveContainer" containerID="5cc98f8276415ff6219c7249dc43ac60e8c0e02385b9396ef777d4946832c5b9"
Sep 18 13:17:45.028823 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:17:45.028807    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cc98f8276415ff6219c7249dc43ac60e8c0e02385b9396ef777d4946832c5b9\": container with ID starting with 5cc98f8276415ff6219c7249dc43ac60e8c0e02385b9396ef777d4946832c5b9 not found: ID does not exist" containerID="5cc98f8276415ff6219c7249dc43ac60e8c0e02385b9396ef777d4946832c5b9"
Sep 18 13:17:45.028863 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.028825    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc98f8276415ff6219c7249dc43ac60e8c0e02385b9396ef777d4946832c5b9"} err="failed to get container status \"5cc98f8276415ff6219c7249dc43ac60e8c0e02385b9396ef777d4946832c5b9\": rpc error: code = NotFound desc = could not find container \"5cc98f8276415ff6219c7249dc43ac60e8c0e02385b9396ef777d4946832c5b9\": container with ID starting with 5cc98f8276415ff6219c7249dc43ac60e8c0e02385b9396ef777d4946832c5b9 not found: ID does not exist"
Sep 18 13:17:45.046498 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046469    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 13:17:45.046756 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046735    2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3da602f1-09f1-4d56-b2c2-659302e87a7d-alertmanager-main-db\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:17:45.046811 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046759    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-main-tls\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:17:45.046811 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046792    2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-web-config\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:17:45.046811 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046801    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3da602f1-09f1-4d56-b2c2-659302e87a7d-metrics-client-ca\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:17:45.046919 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046813    2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3da602f1-09f1-4d56-b2c2-659302e87a7d-tls-assets\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:17:45.046919 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046823    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgk96\" (UniqueName: \"kubernetes.io/projected/3da602f1-09f1-4d56-b2c2-659302e87a7d-kube-api-access-mgk96\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:17:45.046919 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046831    2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3da602f1-09f1-4d56-b2c2-659302e87a7d-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:17:45.046919 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046841    2566 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-config-volume\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:17:45.046919 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046849    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:17:45.046919 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046858    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:17:45.046919 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046866    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:17:45.046919 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046800    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="config-reloader"
Sep 18 13:17:45.046919 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046875    2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3da602f1-09f1-4d56-b2c2-659302e87a7d-config-out\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:17:45.046919 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046881    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="config-reloader"
Sep 18 13:17:45.046919 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046895    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="alertmanager"
Sep 18 13:17:45.046919 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046900    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="alertmanager"
Sep 18 13:17:45.046919 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046910    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bb07b38-9199-424d-9fb7-3ffa57def30a" containerName="registry"
Sep 18 13:17:45.046919 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046915    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bb07b38-9199-424d-9fb7-3ffa57def30a" containerName="registry"
Sep 18 13:17:45.047320 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046931    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="kube-rbac-proxy-web"
Sep 18 13:17:45.047320 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046940    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="kube-rbac-proxy-web"
Sep 18 13:17:45.047320 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046946    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="kube-rbac-proxy"
Sep 18 13:17:45.047320 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046952    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="kube-rbac-proxy"
Sep 18 13:17:45.047320 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046960    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="kube-rbac-proxy-metric"
Sep 18 13:17:45.047320 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046966    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="kube-rbac-proxy-metric"
Sep 18 13:17:45.047320 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046972    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="prom-label-proxy"
Sep 18 13:17:45.047320 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046977    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="prom-label-proxy"
Sep 18 13:17:45.047320 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046985    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="init-config-reloader"
Sep 18 13:17:45.047320 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046990    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="init-config-reloader"
Sep 18 13:17:45.047320 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.046883    2566 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3da602f1-09f1-4d56-b2c2-659302e87a7d-cluster-tls-config\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:17:45.047320 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.047046    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="alertmanager"
Sep 18 13:17:45.047320 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.047053    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="config-reloader"
Sep 18 13:17:45.047320 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.047059    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="kube-rbac-proxy-web"
Sep 18 13:17:45.047320 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.047066    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="kube-rbac-proxy"
Sep 18 13:17:45.047320 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.047073    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="prom-label-proxy"
Sep 18 13:17:45.047320 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.047078    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" containerName="kube-rbac-proxy-metric"
Sep 18 13:17:45.047320 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.047086    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bb07b38-9199-424d-9fb7-3ffa57def30a" containerName="registry"
Sep 18 13:17:45.050568 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.050553    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.055382 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.055359    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 13:17:45.056061 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.055754    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 13:17:45.056061 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.055784    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 13:17:45.056061 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.055791    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 13:17:45.056061 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.055995    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 13:17:45.056331 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.056148    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 13:17:45.056389 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.056353    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 13:17:45.056439 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.056392    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 13:17:45.056485 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.056458    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zkjlc\""
Sep 18 13:17:45.062693 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.062672    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 13:17:45.069345 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.069321    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 13:17:45.147394 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.147356    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-web-config\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.147394 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.147395    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.147589 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.147422    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k2m8\" (UniqueName: \"kubernetes.io/projected/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-kube-api-access-7k2m8\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.147589 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.147456    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.147589 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.147508    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.147589 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.147540    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.147589 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.147559    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.147801 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.147628    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.147801 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.147661    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-config-out\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.147801 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.147678    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.147801 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.147721    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.147801 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.147777    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-config-volume\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.147801 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.147794    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.248546 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.248439    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-config-out\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.248546 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.248479    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.248789 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.248613    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.248789 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.248675    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-config-volume\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.248789 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.248705    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.248789 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.248733    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-web-config\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.248789 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.248766    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.249033 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.248816    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k2m8\" (UniqueName: \"kubernetes.io/projected/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-kube-api-access-7k2m8\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.249033 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.248845    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.249033 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.248880    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.249033 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.248906    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.249033 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.248933    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.249033 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.248971    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.249320 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.249076    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.249742 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.249394    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.251487 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.251461    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-config-out\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.251699 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.251674    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.251778 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.251734    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-config-volume\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.251842 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.251818    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-web-config\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.251948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.251923    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.252046 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.251949    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.252291 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.252262    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.252366 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.252304    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.252538 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.252518    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.253640 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.253621    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.261553 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.261531    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k2m8\" (UniqueName: \"kubernetes.io/projected/ebdcd5ef-4bda-4dd2-a279-6d96c37d011d-kube-api-access-7k2m8\") pod \"alertmanager-main-0\" (UID: \"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.359582 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.359544    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 13:17:45.638885 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.638854    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 13:17:45.641902 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:17:45.641876    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebdcd5ef_4bda_4dd2_a279_6d96c37d011d.slice/crio-018f37ec6292ad61405e09788a0ce70abdb5a1db2e71acf17a282f83679be1d4 WatchSource:0}: Error finding container 018f37ec6292ad61405e09788a0ce70abdb5a1db2e71acf17a282f83679be1d4: Status 404 returned error can't find the container with id 018f37ec6292ad61405e09788a0ce70abdb5a1db2e71acf17a282f83679be1d4
Sep 18 13:17:45.975106 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.975016    2566 generic.go:356] "Generic (PLEG): container finished" podID="ebdcd5ef-4bda-4dd2-a279-6d96c37d011d" containerID="2e2a2560e5abc787ff0de5f93276bfa944d6f1e6936ce8c0b09c3c79133a4852" exitCode=0
Sep 18 13:17:45.975514 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.975105    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d","Type":"ContainerDied","Data":"2e2a2560e5abc787ff0de5f93276bfa944d6f1e6936ce8c0b09c3c79133a4852"}
Sep 18 13:17:45.975514 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:45.975178    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d","Type":"ContainerStarted","Data":"018f37ec6292ad61405e09788a0ce70abdb5a1db2e71acf17a282f83679be1d4"}
Sep 18 13:17:46.079445 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:46.077015    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3da602f1-09f1-4d56-b2c2-659302e87a7d" path="/var/lib/kubelet/pods/3da602f1-09f1-4d56-b2c2-659302e87a7d/volumes"
Sep 18 13:17:46.983587 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:46.983549    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d","Type":"ContainerStarted","Data":"5fbe76b15ab773f5f892384416bb2aef0fda58ff8902c5681fc6c39df9a14517"}
Sep 18 13:17:46.984078 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:46.983646    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d","Type":"ContainerStarted","Data":"848d40766d6d5bcd7b309c8059b79fdd3aa5b43e2ee72c16fb7df6d24a3955d2"}
Sep 18 13:17:46.984078 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:46.983693    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d","Type":"ContainerStarted","Data":"99bd626872c169a31b25d1bac74e6fecf28cda9171a1cd615d4aefabd0f7eba9"}
Sep 18 13:17:46.984078 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:46.983737    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d","Type":"ContainerStarted","Data":"ba066f99867ba7803c0fddafc44f78ebd80cedf84aeb72c32c45adb06d834282"}
Sep 18 13:17:46.984078 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:46.983777    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d","Type":"ContainerStarted","Data":"49ad646d6021b197468308608566aa734490e4469c98ece4ad52949b51467082"}
Sep 18 13:17:46.984078 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:46.983815    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ebdcd5ef-4bda-4dd2-a279-6d96c37d011d","Type":"ContainerStarted","Data":"e4f026f36ca664f0d521fa93883edf27fc3b59f7ad04014a73f00e2f663316c2"}
Sep 18 13:17:47.825296 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.825238    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.810366421 podStartE2EDuration="2.825219313s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.852892ms" isStatefulPod=false podCreationTimestamp="2026-09-18 13:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:17:47.022899835 +0000 UTC m=+175.457776241" watchObservedRunningTime="2026-09-18 13:17:47.825219313 +0000 UTC m=+176.260095716"
Sep 18 13:17:47.825906 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.825883    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5bc57df576-ffwq4"]
Sep 18 13:17:47.828310 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.828290    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:47.830795 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.830765    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 18 13:17:47.830795 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.830776    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 18 13:17:47.830979 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.830776    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 18 13:17:47.831116 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.831097    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kvt8q\""
Sep 18 13:17:47.831263 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.831248    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 18 13:17:47.831443 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.831428    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 18 13:17:47.838406 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.838381    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 18 13:17:47.862248 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.862220    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5bc57df576-ffwq4"]
Sep 18 13:17:47.869929 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.869901    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0-secret-telemeter-client\") pod \"telemeter-client-5bc57df576-ffwq4\" (UID: \"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0\") " pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:47.870046 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.869943    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0-federate-client-tls\") pod \"telemeter-client-5bc57df576-ffwq4\" (UID: \"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0\") " pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:47.870046 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.869975    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0-metrics-client-ca\") pod \"telemeter-client-5bc57df576-ffwq4\" (UID: \"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0\") " pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:47.870046 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.870015    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bc57df576-ffwq4\" (UID: \"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0\") " pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:47.870046 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.870035    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp2k9\" (UniqueName: \"kubernetes.io/projected/147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0-kube-api-access-lp2k9\") pod \"telemeter-client-5bc57df576-ffwq4\" (UID: \"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0\") " pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:47.870214 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.870072    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0-serving-certs-ca-bundle\") pod \"telemeter-client-5bc57df576-ffwq4\" (UID: \"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0\") " pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:47.870214 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.870096    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0-telemeter-client-tls\") pod \"telemeter-client-5bc57df576-ffwq4\" (UID: \"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0\") " pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:47.870214 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.870116    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bc57df576-ffwq4\" (UID: \"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0\") " pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:47.970776 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.970741    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0-federate-client-tls\") pod \"telemeter-client-5bc57df576-ffwq4\" (UID: \"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0\") " pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:47.970776 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.970786    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0-metrics-client-ca\") pod \"telemeter-client-5bc57df576-ffwq4\" (UID: \"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0\") " pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:47.971039 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.970812    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bc57df576-ffwq4\" (UID: \"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0\") " pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:47.971095 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.971049    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp2k9\" (UniqueName: \"kubernetes.io/projected/147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0-kube-api-access-lp2k9\") pod \"telemeter-client-5bc57df576-ffwq4\" (UID: \"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0\") " pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:47.971145 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.971091    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0-serving-certs-ca-bundle\") pod \"telemeter-client-5bc57df576-ffwq4\" (UID: \"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0\") " pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:47.971145 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.971131    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0-telemeter-client-tls\") pod \"telemeter-client-5bc57df576-ffwq4\" (UID: \"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0\") " pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:47.971254 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.971164    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bc57df576-ffwq4\" (UID: \"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0\") " pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:47.971254 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.971213    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0-secret-telemeter-client\") pod \"telemeter-client-5bc57df576-ffwq4\" (UID: \"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0\") " pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:47.971698 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.971571    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0-metrics-client-ca\") pod \"telemeter-client-5bc57df576-ffwq4\" (UID: \"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0\") " pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:47.971813 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.971720    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bc57df576-ffwq4\" (UID: \"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0\") " pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:47.971813 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.971761    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0-serving-certs-ca-bundle\") pod \"telemeter-client-5bc57df576-ffwq4\" (UID: \"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0\") " pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:47.973464 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.973441    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0-federate-client-tls\") pod \"telemeter-client-5bc57df576-ffwq4\" (UID: \"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0\") " pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:47.973637 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.973587    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0-secret-telemeter-client\") pod \"telemeter-client-5bc57df576-ffwq4\" (UID: \"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0\") " pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:47.974065 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.974044    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bc57df576-ffwq4\" (UID: \"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0\") " pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:47.974104 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.974080    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0-telemeter-client-tls\") pod \"telemeter-client-5bc57df576-ffwq4\" (UID: \"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0\") " pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:47.991114 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:47.991090    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp2k9\" (UniqueName: \"kubernetes.io/projected/147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0-kube-api-access-lp2k9\") pod \"telemeter-client-5bc57df576-ffwq4\" (UID: \"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0\") " pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:48.138434 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:48.138384    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4"
Sep 18 13:17:48.435192 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:48.435122    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5bc57df576-ffwq4"]
Sep 18 13:17:48.437525 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:17:48.437497    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147f2b4c_f4a2_4be5_a57b_6b098a8c8dd0.slice/crio-8672b1543f79d9c3484eb4b914db793faad33c76f2670395bed11a12e993d9c8 WatchSource:0}: Error finding container 8672b1543f79d9c3484eb4b914db793faad33c76f2670395bed11a12e993d9c8: Status 404 returned error can't find the container with id 8672b1543f79d9c3484eb4b914db793faad33c76f2670395bed11a12e993d9c8
Sep 18 13:17:48.992691 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:48.992659    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4" event={"ID":"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0","Type":"ContainerStarted","Data":"8672b1543f79d9c3484eb4b914db793faad33c76f2670395bed11a12e993d9c8"}
Sep 18 13:17:51.002587 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:51.002548    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4" event={"ID":"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0","Type":"ContainerStarted","Data":"ef5d22fd2a99bd91f85224522edb4a5601c77f3c2a22113f8d7763cd0ccc9a9e"}
Sep 18 13:17:51.002587 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:51.002631    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4" event={"ID":"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0","Type":"ContainerStarted","Data":"94791fd26b9f8004515788915963555b3c6e2f1e64942bc07673b5fbef0ea7a1"}
Sep 18 13:17:51.002587 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:51.002653    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4" event={"ID":"147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0","Type":"ContainerStarted","Data":"3169169f15e7a99a7cd686d607c458cdc0279c836caa43dd9ae6d4a1b2514370"}
Sep 18 13:17:51.036120 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:17:51.036059    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5bc57df576-ffwq4" podStartSLOduration=2.058722173 podStartE2EDuration="4.036040879s" totalImagesPullingTime="1.977318706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:17:51.035189616 +0000 UTC m=+179.470066019" watchObservedRunningTime="2026-09-18 13:17:51.036040879 +0000 UTC m=+179.470917280"
Sep 18 13:19:51.946215 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:19:51.946187    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k86vn_55e33e14-3f7a-4793-a724-9fd5022740d7/console-operator/1.log"
Sep 18 13:19:51.948442 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:19:51.948419    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k86vn_55e33e14-3f7a-4793-a724-9fd5022740d7/console-operator/1.log"
Sep 18 13:19:51.958113 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:19:51.958089    2566 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 13:20:17.951053 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:17.951019    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kh8sp"]
Sep 18 13:20:17.954131 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:17.954116    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-kh8sp"
Sep 18 13:20:17.956272 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:17.956240    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 18 13:20:17.956799 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:17.956779    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-9dbd7\""
Sep 18 13:20:17.956853 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:17.956798    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 18 13:20:17.969203 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:17.969178    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kh8sp"]
Sep 18 13:20:18.036361 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:18.036319    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/605cd029-744d-4e3d-b959-5f69ab39a236-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kh8sp\" (UID: \"605cd029-744d-4e3d-b959-5f69ab39a236\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kh8sp"
Sep 18 13:20:18.036361 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:18.036360    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgnvk\" (UniqueName: \"kubernetes.io/projected/605cd029-744d-4e3d-b959-5f69ab39a236-kube-api-access-vgnvk\") pod \"cert-manager-webhook-6486ffd997-kh8sp\" (UID: \"605cd029-744d-4e3d-b959-5f69ab39a236\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kh8sp"
Sep 18 13:20:18.136873 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:18.136832    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/605cd029-744d-4e3d-b959-5f69ab39a236-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kh8sp\" (UID: \"605cd029-744d-4e3d-b959-5f69ab39a236\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kh8sp"
Sep 18 13:20:18.136873 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:18.136875    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgnvk\" (UniqueName: \"kubernetes.io/projected/605cd029-744d-4e3d-b959-5f69ab39a236-kube-api-access-vgnvk\") pod \"cert-manager-webhook-6486ffd997-kh8sp\" (UID: \"605cd029-744d-4e3d-b959-5f69ab39a236\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kh8sp"
Sep 18 13:20:18.154427 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:18.154397    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/605cd029-744d-4e3d-b959-5f69ab39a236-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-kh8sp\" (UID: \"605cd029-744d-4e3d-b959-5f69ab39a236\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kh8sp"
Sep 18 13:20:18.154729 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:18.154711    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgnvk\" (UniqueName: \"kubernetes.io/projected/605cd029-744d-4e3d-b959-5f69ab39a236-kube-api-access-vgnvk\") pod \"cert-manager-webhook-6486ffd997-kh8sp\" (UID: \"605cd029-744d-4e3d-b959-5f69ab39a236\") " pod="cert-manager/cert-manager-webhook-6486ffd997-kh8sp"
Sep 18 13:20:18.276112 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:18.276011    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-kh8sp"
Sep 18 13:20:18.547526 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:18.547424    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-kh8sp"]
Sep 18 13:20:18.549689 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:20:18.549663    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod605cd029_744d_4e3d_b959_5f69ab39a236.slice/crio-549958c08b18c8aa167c71e3d0358d5600216cf3f80b15930ae8486e8487229c WatchSource:0}: Error finding container 549958c08b18c8aa167c71e3d0358d5600216cf3f80b15930ae8486e8487229c: Status 404 returned error can't find the container with id 549958c08b18c8aa167c71e3d0358d5600216cf3f80b15930ae8486e8487229c
Sep 18 13:20:18.551457 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:18.551439    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 13:20:18.571072 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:18.571040    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-kh8sp" event={"ID":"605cd029-744d-4e3d-b959-5f69ab39a236","Type":"ContainerStarted","Data":"549958c08b18c8aa167c71e3d0358d5600216cf3f80b15930ae8486e8487229c"}
Sep 18 13:20:22.593440 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:22.593405    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-kh8sp" event={"ID":"605cd029-744d-4e3d-b959-5f69ab39a236","Type":"ContainerStarted","Data":"650d7ca906082eb6c92cc85262e4d8fb8e13c0ef0e196bb4fdb8c25ad4ab8179"}
Sep 18 13:20:22.593941 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:22.593523    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-kh8sp"
Sep 18 13:20:22.622777 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:22.622727    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-kh8sp" podStartSLOduration=2.2916589050000002 podStartE2EDuration="5.622709287s" totalImagesPullingTime="3.331050382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:20:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:20:22.620699018 +0000 UTC m=+331.055575433" watchObservedRunningTime="2026-09-18 13:20:22.622709287 +0000 UTC m=+331.057585691"
Sep 18 13:20:28.599250 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:28.599213    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-kh8sp"
Sep 18 13:20:29.176744 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:29.176715    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7w7qv"]
Sep 18 13:20:29.180020 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:29.179997    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-7w7qv"
Sep 18 13:20:29.183902 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:29.183880    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-s87g8\""
Sep 18 13:20:29.192162 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:29.192134    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7w7qv"]
Sep 18 13:20:29.226668 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:29.226627    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62pcm\" (UniqueName: \"kubernetes.io/projected/2ba28ac6-275f-40db-bb72-d9fafabfe439-kube-api-access-62pcm\") pod \"cert-manager-55b4d954d5-7w7qv\" (UID: \"2ba28ac6-275f-40db-bb72-d9fafabfe439\") " pod="cert-manager/cert-manager-55b4d954d5-7w7qv"
Sep 18 13:20:29.226668 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:29.226676    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ba28ac6-275f-40db-bb72-d9fafabfe439-bound-sa-token\") pod \"cert-manager-55b4d954d5-7w7qv\" (UID: \"2ba28ac6-275f-40db-bb72-d9fafabfe439\") " pod="cert-manager/cert-manager-55b4d954d5-7w7qv"
Sep 18 13:20:29.327637 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:29.327581    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62pcm\" (UniqueName: \"kubernetes.io/projected/2ba28ac6-275f-40db-bb72-d9fafabfe439-kube-api-access-62pcm\") pod \"cert-manager-55b4d954d5-7w7qv\" (UID: \"2ba28ac6-275f-40db-bb72-d9fafabfe439\") " pod="cert-manager/cert-manager-55b4d954d5-7w7qv"
Sep 18 13:20:29.327797 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:29.327646    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ba28ac6-275f-40db-bb72-d9fafabfe439-bound-sa-token\") pod \"cert-manager-55b4d954d5-7w7qv\" (UID: \"2ba28ac6-275f-40db-bb72-d9fafabfe439\") " pod="cert-manager/cert-manager-55b4d954d5-7w7qv"
Sep 18 13:20:29.346234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:29.346210    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ba28ac6-275f-40db-bb72-d9fafabfe439-bound-sa-token\") pod \"cert-manager-55b4d954d5-7w7qv\" (UID: \"2ba28ac6-275f-40db-bb72-d9fafabfe439\") " pod="cert-manager/cert-manager-55b4d954d5-7w7qv"
Sep 18 13:20:29.346336 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:29.346282    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62pcm\" (UniqueName: \"kubernetes.io/projected/2ba28ac6-275f-40db-bb72-d9fafabfe439-kube-api-access-62pcm\") pod \"cert-manager-55b4d954d5-7w7qv\" (UID: \"2ba28ac6-275f-40db-bb72-d9fafabfe439\") " pod="cert-manager/cert-manager-55b4d954d5-7w7qv"
Sep 18 13:20:29.492427 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:29.492338    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-7w7qv"
Sep 18 13:20:29.764436 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:29.764359    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7w7qv"]
Sep 18 13:20:29.767353 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:20:29.767315    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ba28ac6_275f_40db_bb72_d9fafabfe439.slice/crio-ab575f05ade7f197e05b24d2c8d9ddffee1643b09ab6a7409ff609c4fcf4c85d WatchSource:0}: Error finding container ab575f05ade7f197e05b24d2c8d9ddffee1643b09ab6a7409ff609c4fcf4c85d: Status 404 returned error can't find the container with id ab575f05ade7f197e05b24d2c8d9ddffee1643b09ab6a7409ff609c4fcf4c85d
Sep 18 13:20:30.625933 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:30.625894    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-7w7qv" event={"ID":"2ba28ac6-275f-40db-bb72-d9fafabfe439","Type":"ContainerStarted","Data":"2e7e3e707cbdcb7f6918c4721bdee539f0f3e52dfa66f0ebca5d43719983a208"}
Sep 18 13:20:30.625933 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:30.625941    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-7w7qv" event={"ID":"2ba28ac6-275f-40db-bb72-d9fafabfe439","Type":"ContainerStarted","Data":"ab575f05ade7f197e05b24d2c8d9ddffee1643b09ab6a7409ff609c4fcf4c85d"}
Sep 18 13:20:30.642490 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:30.642439    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-7w7qv" podStartSLOduration=1.642422878 podStartE2EDuration="1.642422878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:20:30.640692266 +0000 UTC m=+339.075568670" watchObservedRunningTime="2026-09-18 13:20:30.642422878 +0000 UTC m=+339.077299282"
Sep 18 13:20:56.544239 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:56.544158    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr"]
Sep 18 13:20:56.547010 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:56.546990    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr"
Sep 18 13:20:56.550803 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:56.550780    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 18 13:20:56.550803 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:56.550790    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 18 13:20:56.550987 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:56.550809    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 18 13:20:56.550987 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:56.550847    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 18 13:20:56.553527 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:56.553239    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-krfj5\""
Sep 18 13:20:56.570611 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:56.570560    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr"]
Sep 18 13:20:56.642974 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:56.642935    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvc6j\" (UniqueName: \"kubernetes.io/projected/ddec8697-ca0a-4402-89cb-f0707b414d64-kube-api-access-hvc6j\") pod \"opendatahub-operator-controller-manager-854585bc8d-zllbr\" (UID: \"ddec8697-ca0a-4402-89cb-f0707b414d64\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr"
Sep 18 13:20:56.643182 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:56.643045    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ddec8697-ca0a-4402-89cb-f0707b414d64-apiservice-cert\") pod \"opendatahub-operator-controller-manager-854585bc8d-zllbr\" (UID: \"ddec8697-ca0a-4402-89cb-f0707b414d64\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr"
Sep 18 13:20:56.643182 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:56.643081    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ddec8697-ca0a-4402-89cb-f0707b414d64-manifests\") pod \"opendatahub-operator-controller-manager-854585bc8d-zllbr\" (UID: \"ddec8697-ca0a-4402-89cb-f0707b414d64\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr"
Sep 18 13:20:56.643292 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:56.643181    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ddec8697-ca0a-4402-89cb-f0707b414d64-webhook-cert\") pod \"opendatahub-operator-controller-manager-854585bc8d-zllbr\" (UID: \"ddec8697-ca0a-4402-89cb-f0707b414d64\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr"
Sep 18 13:20:56.643292 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:56.643235    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddec8697-ca0a-4402-89cb-f0707b414d64-tmp\") pod \"opendatahub-operator-controller-manager-854585bc8d-zllbr\" (UID: \"ddec8697-ca0a-4402-89cb-f0707b414d64\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr"
Sep 18 13:20:56.744523 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:56.744485    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvc6j\" (UniqueName: \"kubernetes.io/projected/ddec8697-ca0a-4402-89cb-f0707b414d64-kube-api-access-hvc6j\") pod \"opendatahub-operator-controller-manager-854585bc8d-zllbr\" (UID: \"ddec8697-ca0a-4402-89cb-f0707b414d64\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr"
Sep 18 13:20:56.744758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:56.744539    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ddec8697-ca0a-4402-89cb-f0707b414d64-apiservice-cert\") pod \"opendatahub-operator-controller-manager-854585bc8d-zllbr\" (UID: \"ddec8697-ca0a-4402-89cb-f0707b414d64\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr"
Sep 18 13:20:56.744758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:56.744560    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ddec8697-ca0a-4402-89cb-f0707b414d64-manifests\") pod \"opendatahub-operator-controller-manager-854585bc8d-zllbr\" (UID: \"ddec8697-ca0a-4402-89cb-f0707b414d64\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr"
Sep 18 13:20:56.744758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:56.744590    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ddec8697-ca0a-4402-89cb-f0707b414d64-webhook-cert\") pod \"opendatahub-operator-controller-manager-854585bc8d-zllbr\" (UID: \"ddec8697-ca0a-4402-89cb-f0707b414d64\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr"
Sep 18 13:20:56.744758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:56.744657    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddec8697-ca0a-4402-89cb-f0707b414d64-tmp\") pod \"opendatahub-operator-controller-manager-854585bc8d-zllbr\" (UID: \"ddec8697-ca0a-4402-89cb-f0707b414d64\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr"
Sep 18 13:20:56.745078 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:56.745054    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ddec8697-ca0a-4402-89cb-f0707b414d64-manifests\") pod \"opendatahub-operator-controller-manager-854585bc8d-zllbr\" (UID: \"ddec8697-ca0a-4402-89cb-f0707b414d64\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr"
Sep 18 13:20:56.747496 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:56.747475    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddec8697-ca0a-4402-89cb-f0707b414d64-tmp\") pod \"opendatahub-operator-controller-manager-854585bc8d-zllbr\" (UID: \"ddec8697-ca0a-4402-89cb-f0707b414d64\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr"
Sep 18 13:20:56.747640 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:56.747619    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ddec8697-ca0a-4402-89cb-f0707b414d64-webhook-cert\") pod \"opendatahub-operator-controller-manager-854585bc8d-zllbr\" (UID: \"ddec8697-ca0a-4402-89cb-f0707b414d64\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr"
Sep 18 13:20:56.747715 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:56.747701    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ddec8697-ca0a-4402-89cb-f0707b414d64-apiservice-cert\") pod \"opendatahub-operator-controller-manager-854585bc8d-zllbr\" (UID: \"ddec8697-ca0a-4402-89cb-f0707b414d64\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr"
Sep 18 13:20:56.753388 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:56.753365    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvc6j\" (UniqueName: \"kubernetes.io/projected/ddec8697-ca0a-4402-89cb-f0707b414d64-kube-api-access-hvc6j\") pod \"opendatahub-operator-controller-manager-854585bc8d-zllbr\" (UID: \"ddec8697-ca0a-4402-89cb-f0707b414d64\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr"
Sep 18 13:20:56.857542 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:56.857503    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr"
Sep 18 13:20:57.163984 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:57.163900    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr"]
Sep 18 13:20:57.167120 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:20:57.167093    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddec8697_ca0a_4402_89cb_f0707b414d64.slice/crio-0dcb46f4b75576ee493e1f65a6827045608ec9ffbcce2f92a153183c1c0d6a61 WatchSource:0}: Error finding container 0dcb46f4b75576ee493e1f65a6827045608ec9ffbcce2f92a153183c1c0d6a61: Status 404 returned error can't find the container with id 0dcb46f4b75576ee493e1f65a6827045608ec9ffbcce2f92a153183c1c0d6a61
Sep 18 13:20:57.734081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:20:57.734037    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr" event={"ID":"ddec8697-ca0a-4402-89cb-f0707b414d64","Type":"ContainerStarted","Data":"0dcb46f4b75576ee493e1f65a6827045608ec9ffbcce2f92a153183c1c0d6a61"}
Sep 18 13:21:00.750704 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:00.750668    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr" event={"ID":"ddec8697-ca0a-4402-89cb-f0707b414d64","Type":"ContainerStarted","Data":"247726d4aa23ee6c2d6845f994831740e249d95b1046ab011e27d4f9d4355be5"}
Sep 18 13:21:01.756432 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:01.756330    2566 generic.go:356] "Generic (PLEG): container finished" podID="ddec8697-ca0a-4402-89cb-f0707b414d64" containerID="247726d4aa23ee6c2d6845f994831740e249d95b1046ab011e27d4f9d4355be5" exitCode=0
Sep 18 13:21:01.756870 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:01.756423    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr" event={"ID":"ddec8697-ca0a-4402-89cb-f0707b414d64","Type":"ContainerDied","Data":"247726d4aa23ee6c2d6845f994831740e249d95b1046ab011e27d4f9d4355be5"}
Sep 18 13:21:02.763159 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:02.763125    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr" event={"ID":"ddec8697-ca0a-4402-89cb-f0707b414d64","Type":"ContainerStarted","Data":"82d6fcddbe4ce2673b773d5b4aee8d46433f9953fc44a37e11cf1559b4c8bdaf"}
Sep 18 13:21:02.763667 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:02.763262    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr"
Sep 18 13:21:02.785079 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:02.785026    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr" podStartSLOduration=2.298578796 podStartE2EDuration="6.785009821s" totalImagesPullingTime="3.389290887s" totalInitContainerRuntime="1.097140138s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:20:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:21:02.783910043 +0000 UTC m=+371.218786447" watchObservedRunningTime="2026-09-18 13:21:02.785009821 +0000 UTC m=+371.219886225"
Sep 18 13:21:13.770679 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:13.770641    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-zllbr"
Sep 18 13:21:16.372508 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:16.372472    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c5fc7c97f-f7z25"]
Sep 18 13:21:16.374947 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:16.374931    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c5fc7c97f-f7z25"
Sep 18 13:21:16.382725 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:16.382699    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 18 13:21:16.383343 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:16.383323    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 13:21:16.383448 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:16.383390    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9d854\""
Sep 18 13:21:16.383448 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:16.383425    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 18 13:21:16.388279 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:16.388262    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 13:21:16.408725 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:16.408691    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c5fc7c97f-f7z25"]
Sep 18 13:21:16.513527 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:16.513486    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7399cabd-9c15-41c4-aef1-cdc23e893e43-tls-certs\") pod \"kube-auth-proxy-7c5fc7c97f-f7z25\" (UID: \"7399cabd-9c15-41c4-aef1-cdc23e893e43\") " pod="openshift-ingress/kube-auth-proxy-7c5fc7c97f-f7z25"
Sep 18 13:21:16.513734 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:16.513540    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7399cabd-9c15-41c4-aef1-cdc23e893e43-tmp\") pod \"kube-auth-proxy-7c5fc7c97f-f7z25\" (UID: \"7399cabd-9c15-41c4-aef1-cdc23e893e43\") " pod="openshift-ingress/kube-auth-proxy-7c5fc7c97f-f7z25"
Sep 18 13:21:16.513734 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:16.513620    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2krs6\" (UniqueName: \"kubernetes.io/projected/7399cabd-9c15-41c4-aef1-cdc23e893e43-kube-api-access-2krs6\") pod \"kube-auth-proxy-7c5fc7c97f-f7z25\" (UID: \"7399cabd-9c15-41c4-aef1-cdc23e893e43\") " pod="openshift-ingress/kube-auth-proxy-7c5fc7c97f-f7z25"
Sep 18 13:21:16.614747 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:16.614711    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7399cabd-9c15-41c4-aef1-cdc23e893e43-tls-certs\") pod \"kube-auth-proxy-7c5fc7c97f-f7z25\" (UID: \"7399cabd-9c15-41c4-aef1-cdc23e893e43\") " pod="openshift-ingress/kube-auth-proxy-7c5fc7c97f-f7z25"
Sep 18 13:21:16.614881 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:16.614767    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7399cabd-9c15-41c4-aef1-cdc23e893e43-tmp\") pod \"kube-auth-proxy-7c5fc7c97f-f7z25\" (UID: \"7399cabd-9c15-41c4-aef1-cdc23e893e43\") " pod="openshift-ingress/kube-auth-proxy-7c5fc7c97f-f7z25"
Sep 18 13:21:16.614881 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:16.614793    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2krs6\" (UniqueName: \"kubernetes.io/projected/7399cabd-9c15-41c4-aef1-cdc23e893e43-kube-api-access-2krs6\") pod \"kube-auth-proxy-7c5fc7c97f-f7z25\" (UID: \"7399cabd-9c15-41c4-aef1-cdc23e893e43\") " pod="openshift-ingress/kube-auth-proxy-7c5fc7c97f-f7z25"
Sep 18 13:21:16.617202 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:16.617178    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7399cabd-9c15-41c4-aef1-cdc23e893e43-tmp\") pod \"kube-auth-proxy-7c5fc7c97f-f7z25\" (UID: \"7399cabd-9c15-41c4-aef1-cdc23e893e43\") " pod="openshift-ingress/kube-auth-proxy-7c5fc7c97f-f7z25"
Sep 18 13:21:16.617338 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:16.617320    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7399cabd-9c15-41c4-aef1-cdc23e893e43-tls-certs\") pod \"kube-auth-proxy-7c5fc7c97f-f7z25\" (UID: \"7399cabd-9c15-41c4-aef1-cdc23e893e43\") " pod="openshift-ingress/kube-auth-proxy-7c5fc7c97f-f7z25"
Sep 18 13:21:16.642278 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:16.642204    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2krs6\" (UniqueName: \"kubernetes.io/projected/7399cabd-9c15-41c4-aef1-cdc23e893e43-kube-api-access-2krs6\") pod \"kube-auth-proxy-7c5fc7c97f-f7z25\" (UID: \"7399cabd-9c15-41c4-aef1-cdc23e893e43\") " pod="openshift-ingress/kube-auth-proxy-7c5fc7c97f-f7z25"
Sep 18 13:21:16.684827 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:16.684788    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c5fc7c97f-f7z25"
Sep 18 13:21:16.963897 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:16.963872    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c5fc7c97f-f7z25"]
Sep 18 13:21:16.965798 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:21:16.965770    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7399cabd_9c15_41c4_aef1_cdc23e893e43.slice/crio-85140c754ea7476794eeaf2ea4c961f6325e821163ccb5a1fda4b2d88ff9ba14 WatchSource:0}: Error finding container 85140c754ea7476794eeaf2ea4c961f6325e821163ccb5a1fda4b2d88ff9ba14: Status 404 returned error can't find the container with id 85140c754ea7476794eeaf2ea4c961f6325e821163ccb5a1fda4b2d88ff9ba14
Sep 18 13:21:17.833182 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:17.833139    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c5fc7c97f-f7z25" event={"ID":"7399cabd-9c15-41c4-aef1-cdc23e893e43","Type":"ContainerStarted","Data":"85140c754ea7476794eeaf2ea4c961f6325e821163ccb5a1fda4b2d88ff9ba14"}
Sep 18 13:21:20.851650 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:20.851615    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c5fc7c97f-f7z25" event={"ID":"7399cabd-9c15-41c4-aef1-cdc23e893e43","Type":"ContainerStarted","Data":"8d5649c871c9ec2142c0509d7d5e8b3e4f386e813af4693f6c3cf97e5390d1c6"}
Sep 18 13:21:20.890100 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:20.890056    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c5fc7c97f-f7z25" podStartSLOduration=1.88800985 podStartE2EDuration="4.890041486s" totalImagesPullingTime="3.002031636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:21:20.884799774 +0000 UTC m=+389.319676178" watchObservedRunningTime="2026-09-18 13:21:20.890041486 +0000 UTC m=+389.324917890"
Sep 18 13:21:24.312085 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.312051    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"]
Sep 18 13:21:24.314678 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.314658    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.317251 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.317224    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 13:21:24.317367 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.317224    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-d57zc\""
Sep 18 13:21:24.339638 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.339611    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"]
Sep 18 13:21:24.376067 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.376031    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.376210 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.376073    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zf7p\" (UniqueName: \"kubernetes.io/projected/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-kube-api-access-8zf7p\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.376210 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.376104    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.376210 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.376124    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.376210 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.376186    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.376341 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.376231    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.376341 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.376251    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.376341 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.376298    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.376341 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.376325    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.476779 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.476736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.476779 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.476782    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.477021 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.476803    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.477021 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.476823    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.477021 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.476842    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.477021 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.476873    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.477221 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.477062    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zf7p\" (UniqueName: \"kubernetes.io/projected/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-kube-api-access-8zf7p\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.477221 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.477140    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.477221 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.477174    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.477221 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.477189    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.477423 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.477330    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.477423 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.477371    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.477561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.477542    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.477988 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.477965    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.479156 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.479138    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.479497 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.479472    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.487741 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.487707    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.487917 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.487899    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zf7p\" (UniqueName: \"kubernetes.io/projected/a6530f20-b9e0-4f5b-bae3-cc99a01845e4-kube-api-access-8zf7p\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4\" (UID: \"a6530f20-b9e0-4f5b-bae3-cc99a01845e4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.624228 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.624190    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:24.903239 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:24.903161    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"]
Sep 18 13:21:24.906344 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:21:24.906312    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6530f20_b9e0_4f5b_bae3_cc99a01845e4.slice/crio-34fcc38e4b73d6d571ab8b9f9443ba72993a24dc149dc64873a2121cdc653f78 WatchSource:0}: Error finding container 34fcc38e4b73d6d571ab8b9f9443ba72993a24dc149dc64873a2121cdc653f78: Status 404 returned error can't find the container with id 34fcc38e4b73d6d571ab8b9f9443ba72993a24dc149dc64873a2121cdc653f78
Sep 18 13:21:25.878285 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:25.878249    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4" event={"ID":"a6530f20-b9e0-4f5b-bae3-cc99a01845e4","Type":"ContainerStarted","Data":"34fcc38e4b73d6d571ab8b9f9443ba72993a24dc149dc64873a2121cdc653f78"}
Sep 18 13:21:27.654220 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:27.654181    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:21:27.654497 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:27.654274    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:21:27.654497 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:27.654302    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:21:27.889003 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:27.888963    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4" event={"ID":"a6530f20-b9e0-4f5b-bae3-cc99a01845e4","Type":"ContainerStarted","Data":"39609d7b591c3a44f837c53b70a8af64d1dbecf56c206b81b493b8d2ef4ace58"}
Sep 18 13:21:27.904766 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:27.904686    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zc529"]
Sep 18 13:21:27.907217 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:27.907196    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-zc529"
Sep 18 13:21:27.909996 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:27.909977    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-7kv2s\""
Sep 18 13:21:27.910169 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:27.910141    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 18 13:21:27.920187 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:27.920140    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4" podStartSLOduration=1.174617571 podStartE2EDuration="3.920120222s" totalImagesPullingTime="2.745502651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:21:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:21:27.917691159 +0000 UTC m=+396.352567793" watchObservedRunningTime="2026-09-18 13:21:27.920120222 +0000 UTC m=+396.354996626"
Sep 18 13:21:27.924286 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:27.924263    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zc529"]
Sep 18 13:21:28.006984 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:28.006944    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p78nx\" (UniqueName: \"kubernetes.io/projected/05bcebef-a0aa-46ec-9bf5-ca9c83c1770b-kube-api-access-p78nx\") pod \"odh-model-controller-7998c67dc4-zc529\" (UID: \"05bcebef-a0aa-46ec-9bf5-ca9c83c1770b\") " pod="opendatahub/odh-model-controller-7998c67dc4-zc529"
Sep 18 13:21:28.007172 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:28.007124    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05bcebef-a0aa-46ec-9bf5-ca9c83c1770b-cert\") pod \"odh-model-controller-7998c67dc4-zc529\" (UID: \"05bcebef-a0aa-46ec-9bf5-ca9c83c1770b\") " pod="opendatahub/odh-model-controller-7998c67dc4-zc529"
Sep 18 13:21:28.108203 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:28.108038    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05bcebef-a0aa-46ec-9bf5-ca9c83c1770b-cert\") pod \"odh-model-controller-7998c67dc4-zc529\" (UID: \"05bcebef-a0aa-46ec-9bf5-ca9c83c1770b\") " pod="opendatahub/odh-model-controller-7998c67dc4-zc529"
Sep 18 13:21:28.108203 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:28.108100    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p78nx\" (UniqueName: \"kubernetes.io/projected/05bcebef-a0aa-46ec-9bf5-ca9c83c1770b-kube-api-access-p78nx\") pod \"odh-model-controller-7998c67dc4-zc529\" (UID: \"05bcebef-a0aa-46ec-9bf5-ca9c83c1770b\") " pod="opendatahub/odh-model-controller-7998c67dc4-zc529"
Sep 18 13:21:28.108467 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:21:28.108239    2566 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 18 13:21:28.108467 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:21:28.108324    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05bcebef-a0aa-46ec-9bf5-ca9c83c1770b-cert podName:05bcebef-a0aa-46ec-9bf5-ca9c83c1770b nodeName:}" failed. No retries permitted until 2026-09-18 13:21:28.608301848 +0000 UTC m=+397.043178237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/05bcebef-a0aa-46ec-9bf5-ca9c83c1770b-cert") pod "odh-model-controller-7998c67dc4-zc529" (UID: "05bcebef-a0aa-46ec-9bf5-ca9c83c1770b") : secret "odh-model-controller-webhook-cert" not found
Sep 18 13:21:28.120579 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:28.120552    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p78nx\" (UniqueName: \"kubernetes.io/projected/05bcebef-a0aa-46ec-9bf5-ca9c83c1770b-kube-api-access-p78nx\") pod \"odh-model-controller-7998c67dc4-zc529\" (UID: \"05bcebef-a0aa-46ec-9bf5-ca9c83c1770b\") " pod="opendatahub/odh-model-controller-7998c67dc4-zc529"
Sep 18 13:21:28.477584 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:28.477543    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-sx4ff"]
Sep 18 13:21:28.480317 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:28.480293    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-sx4ff"
Sep 18 13:21:28.483428 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:28.483400    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 18 13:21:28.483555 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:28.483464    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-dgw6f\""
Sep 18 13:21:28.501481 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:28.501453    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-sx4ff"]
Sep 18 13:21:28.511442 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:28.511418    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brmxs\" (UniqueName: \"kubernetes.io/projected/9b4bc476-98f6-4663-a919-1351827790dd-kube-api-access-brmxs\") pod \"model-serving-api-54d89bd5fc-sx4ff\" (UID: \"9b4bc476-98f6-4663-a919-1351827790dd\") " pod="opendatahub/model-serving-api-54d89bd5fc-sx4ff"
Sep 18 13:21:28.511567 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:28.511529    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b4bc476-98f6-4663-a919-1351827790dd-tls-certs\") pod \"model-serving-api-54d89bd5fc-sx4ff\" (UID: \"9b4bc476-98f6-4663-a919-1351827790dd\") " pod="opendatahub/model-serving-api-54d89bd5fc-sx4ff"
Sep 18 13:21:28.612580 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:28.612543    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05bcebef-a0aa-46ec-9bf5-ca9c83c1770b-cert\") pod \"odh-model-controller-7998c67dc4-zc529\" (UID: \"05bcebef-a0aa-46ec-9bf5-ca9c83c1770b\") " pod="opendatahub/odh-model-controller-7998c67dc4-zc529"
Sep 18 13:21:28.612580 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:28.612578    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b4bc476-98f6-4663-a919-1351827790dd-tls-certs\") pod \"model-serving-api-54d89bd5fc-sx4ff\" (UID: \"9b4bc476-98f6-4663-a919-1351827790dd\") " pod="opendatahub/model-serving-api-54d89bd5fc-sx4ff"
Sep 18 13:21:28.612804 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:28.612646    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brmxs\" (UniqueName: \"kubernetes.io/projected/9b4bc476-98f6-4663-a919-1351827790dd-kube-api-access-brmxs\") pod \"model-serving-api-54d89bd5fc-sx4ff\" (UID: \"9b4bc476-98f6-4663-a919-1351827790dd\") " pod="opendatahub/model-serving-api-54d89bd5fc-sx4ff"
Sep 18 13:21:28.615011 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:28.614987    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b4bc476-98f6-4663-a919-1351827790dd-tls-certs\") pod \"model-serving-api-54d89bd5fc-sx4ff\" (UID: \"9b4bc476-98f6-4663-a919-1351827790dd\") " pod="opendatahub/model-serving-api-54d89bd5fc-sx4ff"
Sep 18 13:21:28.615146 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:28.615124    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05bcebef-a0aa-46ec-9bf5-ca9c83c1770b-cert\") pod \"odh-model-controller-7998c67dc4-zc529\" (UID: \"05bcebef-a0aa-46ec-9bf5-ca9c83c1770b\") " pod="opendatahub/odh-model-controller-7998c67dc4-zc529"
Sep 18 13:21:28.621155 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:28.621126    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brmxs\" (UniqueName: \"kubernetes.io/projected/9b4bc476-98f6-4663-a919-1351827790dd-kube-api-access-brmxs\") pod \"model-serving-api-54d89bd5fc-sx4ff\" (UID: \"9b4bc476-98f6-4663-a919-1351827790dd\") " pod="opendatahub/model-serving-api-54d89bd5fc-sx4ff"
Sep 18 13:21:28.624823 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:28.624799    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:28.629346 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:28.629327    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:28.791006 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:28.790914    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-sx4ff"
Sep 18 13:21:28.817197 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:28.817165    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-zc529"
Sep 18 13:21:28.893725 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:28.893696    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:28.894785 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:28.894765    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4"
Sep 18 13:21:29.075904 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:29.075856    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-sx4ff"]
Sep 18 13:21:29.078283 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:21:29.078252    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4bc476_98f6_4663_a919_1351827790dd.slice/crio-458de0ec7f7cf798ccf121ed6d43407b400a7d612e59e0b7fc9cada3ccb58272 WatchSource:0}: Error finding container 458de0ec7f7cf798ccf121ed6d43407b400a7d612e59e0b7fc9cada3ccb58272: Status 404 returned error can't find the container with id 458de0ec7f7cf798ccf121ed6d43407b400a7d612e59e0b7fc9cada3ccb58272
Sep 18 13:21:29.096453 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:29.096430    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zc529"]
Sep 18 13:21:29.098865 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:21:29.098838    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05bcebef_a0aa_46ec_9bf5_ca9c83c1770b.slice/crio-cef440ac4ada3ba5deda2c77b82ac1ee47d2d4df75c78eea43735740382cc714 WatchSource:0}: Error finding container cef440ac4ada3ba5deda2c77b82ac1ee47d2d4df75c78eea43735740382cc714: Status 404 returned error can't find the container with id cef440ac4ada3ba5deda2c77b82ac1ee47d2d4df75c78eea43735740382cc714
Sep 18 13:21:29.901584 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:29.901521    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-zc529" event={"ID":"05bcebef-a0aa-46ec-9bf5-ca9c83c1770b","Type":"ContainerStarted","Data":"cef440ac4ada3ba5deda2c77b82ac1ee47d2d4df75c78eea43735740382cc714"}
Sep 18 13:21:29.903464 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:29.903431    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-sx4ff" event={"ID":"9b4bc476-98f6-4663-a919-1351827790dd","Type":"ContainerStarted","Data":"458de0ec7f7cf798ccf121ed6d43407b400a7d612e59e0b7fc9cada3ccb58272"}
Sep 18 13:21:32.927727 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:32.927682    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-zc529" event={"ID":"05bcebef-a0aa-46ec-9bf5-ca9c83c1770b","Type":"ContainerStarted","Data":"35b313ec506ca290a730aba55ce1e9a1f30109120e838798b5c0615bf7a50a73"}
Sep 18 13:21:32.928141 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:32.927830    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-zc529"
Sep 18 13:21:32.929266 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:32.929239    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-sx4ff" event={"ID":"9b4bc476-98f6-4663-a919-1351827790dd","Type":"ContainerStarted","Data":"4b1e5279fd5dfca16f2ca9c8e61f8afb4795e526622a443c654d1c09d3818928"}
Sep 18 13:21:32.929364 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:32.929304    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-sx4ff"
Sep 18 13:21:32.947114 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:32.947070    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-zc529" podStartSLOduration=2.8598432049999998 podStartE2EDuration="5.94705539s" totalImagesPullingTime="3.087212185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:21:32.945832063 +0000 UTC m=+401.380708469" watchObservedRunningTime="2026-09-18 13:21:32.94705539 +0000 UTC m=+401.381931796"
Sep 18 13:21:32.966890 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:32.966847    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-sx4ff" podStartSLOduration=1.894360408 podStartE2EDuration="4.966832154s" totalImagesPullingTime="3.072471746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:21:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:21:32.965094932 +0000 UTC m=+401.399971335" watchObservedRunningTime="2026-09-18 13:21:32.966832154 +0000 UTC m=+401.401708558"
Sep 18 13:21:34.925386 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:34.925349    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"]
Sep 18 13:21:34.930146 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:34.930116    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:34.933729 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:34.933687    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 18 13:21:34.933887 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:34.933861    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 18 13:21:34.933955 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:34.933925    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 18 13:21:34.934037 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:34.934020    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-c66ql\""
Sep 18 13:21:34.955558 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:34.955530    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"]
Sep 18 13:21:34.968374 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:34.968349    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c4f346-cfd7-4e0d-b482-2f670e7817d4-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-8lgh8\" (UID: \"a8c4f346-cfd7-4e0d-b482-2f670e7817d4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:34.968374 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:34.968379    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a8c4f346-cfd7-4e0d-b482-2f670e7817d4-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-8lgh8\" (UID: \"a8c4f346-cfd7-4e0d-b482-2f670e7817d4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:34.968615 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:34.968404    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c4f346-cfd7-4e0d-b482-2f670e7817d4-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-8lgh8\" (UID: \"a8c4f346-cfd7-4e0d-b482-2f670e7817d4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:34.968615 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:34.968479    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jzhg\" (UniqueName: \"kubernetes.io/projected/a8c4f346-cfd7-4e0d-b482-2f670e7817d4-kube-api-access-4jzhg\") pod \"istiod-openshift-gateway-5677ccb49b-8lgh8\" (UID: \"a8c4f346-cfd7-4e0d-b482-2f670e7817d4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:34.968615 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:34.968529    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a8c4f346-cfd7-4e0d-b482-2f670e7817d4-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-8lgh8\" (UID: \"a8c4f346-cfd7-4e0d-b482-2f670e7817d4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:34.968615 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:34.968561    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a8c4f346-cfd7-4e0d-b482-2f670e7817d4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-8lgh8\" (UID: \"a8c4f346-cfd7-4e0d-b482-2f670e7817d4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:34.968792 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:34.968644    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a8c4f346-cfd7-4e0d-b482-2f670e7817d4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-8lgh8\" (UID: \"a8c4f346-cfd7-4e0d-b482-2f670e7817d4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:35.069716 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:35.069674    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jzhg\" (UniqueName: \"kubernetes.io/projected/a8c4f346-cfd7-4e0d-b482-2f670e7817d4-kube-api-access-4jzhg\") pod \"istiod-openshift-gateway-5677ccb49b-8lgh8\" (UID: \"a8c4f346-cfd7-4e0d-b482-2f670e7817d4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:35.069716 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:35.069722    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a8c4f346-cfd7-4e0d-b482-2f670e7817d4-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-8lgh8\" (UID: \"a8c4f346-cfd7-4e0d-b482-2f670e7817d4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:35.069973 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:35.069745    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a8c4f346-cfd7-4e0d-b482-2f670e7817d4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-8lgh8\" (UID: \"a8c4f346-cfd7-4e0d-b482-2f670e7817d4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:35.069973 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:35.069776    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a8c4f346-cfd7-4e0d-b482-2f670e7817d4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-8lgh8\" (UID: \"a8c4f346-cfd7-4e0d-b482-2f670e7817d4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:35.069973 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:35.069835    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c4f346-cfd7-4e0d-b482-2f670e7817d4-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-8lgh8\" (UID: \"a8c4f346-cfd7-4e0d-b482-2f670e7817d4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:35.069973 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:35.069862    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a8c4f346-cfd7-4e0d-b482-2f670e7817d4-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-8lgh8\" (UID: \"a8c4f346-cfd7-4e0d-b482-2f670e7817d4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:35.069973 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:35.069906    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c4f346-cfd7-4e0d-b482-2f670e7817d4-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-8lgh8\" (UID: \"a8c4f346-cfd7-4e0d-b482-2f670e7817d4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:35.070633 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:35.070581    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a8c4f346-cfd7-4e0d-b482-2f670e7817d4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-8lgh8\" (UID: \"a8c4f346-cfd7-4e0d-b482-2f670e7817d4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:35.072255 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:35.072233    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a8c4f346-cfd7-4e0d-b482-2f670e7817d4-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-8lgh8\" (UID: \"a8c4f346-cfd7-4e0d-b482-2f670e7817d4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:35.072311 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:35.072275    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c4f346-cfd7-4e0d-b482-2f670e7817d4-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-8lgh8\" (UID: \"a8c4f346-cfd7-4e0d-b482-2f670e7817d4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:35.072640 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:35.072620    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a8c4f346-cfd7-4e0d-b482-2f670e7817d4-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-8lgh8\" (UID: \"a8c4f346-cfd7-4e0d-b482-2f670e7817d4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:35.072684 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:35.072625    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a8c4f346-cfd7-4e0d-b482-2f670e7817d4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-8lgh8\" (UID: \"a8c4f346-cfd7-4e0d-b482-2f670e7817d4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:35.077475 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:35.077453    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c4f346-cfd7-4e0d-b482-2f670e7817d4-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-8lgh8\" (UID: \"a8c4f346-cfd7-4e0d-b482-2f670e7817d4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:35.078239 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:35.078218    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jzhg\" (UniqueName: \"kubernetes.io/projected/a8c4f346-cfd7-4e0d-b482-2f670e7817d4-kube-api-access-4jzhg\") pod \"istiod-openshift-gateway-5677ccb49b-8lgh8\" (UID: \"a8c4f346-cfd7-4e0d-b482-2f670e7817d4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:35.240782 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:35.240687    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:35.521535 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:35.521460    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"]
Sep 18 13:21:35.525370 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:21:35.525341    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c4f346_cfd7_4e0d_b482_2f670e7817d4.slice/crio-b11b4347766033ec6696079834f81f3f5daa789c022d07fd92dd702e8bfb9216 WatchSource:0}: Error finding container b11b4347766033ec6696079834f81f3f5daa789c022d07fd92dd702e8bfb9216: Status 404 returned error can't find the container with id b11b4347766033ec6696079834f81f3f5daa789c022d07fd92dd702e8bfb9216
Sep 18 13:21:35.944318 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:35.944285    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8" event={"ID":"a8c4f346-cfd7-4e0d-b482-2f670e7817d4","Type":"ContainerStarted","Data":"b11b4347766033ec6696079834f81f3f5daa789c022d07fd92dd702e8bfb9216"}
Sep 18 13:21:36.656206 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:36.656173    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-85985676b5-4gsh6"]
Sep 18 13:21:36.658759 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:36.658736    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85985676b5-4gsh6"
Sep 18 13:21:36.662486 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:36.662460    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 13:21:36.662612 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:36.662571    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-n2mgq\""
Sep 18 13:21:36.663337 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:36.663315    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 13:21:36.665746 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:36.665690    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 13:21:36.667558 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:36.667470    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:21:36.673328 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:36.673310    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 13:21:36.678539 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:36.678517    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85985676b5-4gsh6"]
Sep 18 13:21:36.682691 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:36.682668    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ed304aa-a520-4060-ab6a-f706761b322c-metrics-cert\") pod \"lws-controller-manager-85985676b5-4gsh6\" (UID: \"7ed304aa-a520-4060-ab6a-f706761b322c\") " pod="openshift-lws-operator/lws-controller-manager-85985676b5-4gsh6"
Sep 18 13:21:36.682807 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:36.682724    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7ed304aa-a520-4060-ab6a-f706761b322c-manager-config\") pod \"lws-controller-manager-85985676b5-4gsh6\" (UID: \"7ed304aa-a520-4060-ab6a-f706761b322c\") " pod="openshift-lws-operator/lws-controller-manager-85985676b5-4gsh6"
Sep 18 13:21:36.682807 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:36.682749    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5tft\" (UniqueName: \"kubernetes.io/projected/7ed304aa-a520-4060-ab6a-f706761b322c-kube-api-access-j5tft\") pod \"lws-controller-manager-85985676b5-4gsh6\" (UID: \"7ed304aa-a520-4060-ab6a-f706761b322c\") " pod="openshift-lws-operator/lws-controller-manager-85985676b5-4gsh6"
Sep 18 13:21:36.682807 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:36.682781    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ed304aa-a520-4060-ab6a-f706761b322c-cert\") pod \"lws-controller-manager-85985676b5-4gsh6\" (UID: \"7ed304aa-a520-4060-ab6a-f706761b322c\") " pod="openshift-lws-operator/lws-controller-manager-85985676b5-4gsh6"
Sep 18 13:21:36.783279 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:36.783243    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7ed304aa-a520-4060-ab6a-f706761b322c-manager-config\") pod \"lws-controller-manager-85985676b5-4gsh6\" (UID: \"7ed304aa-a520-4060-ab6a-f706761b322c\") " pod="openshift-lws-operator/lws-controller-manager-85985676b5-4gsh6"
Sep 18 13:21:36.783279 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:36.783282    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5tft\" (UniqueName: \"kubernetes.io/projected/7ed304aa-a520-4060-ab6a-f706761b322c-kube-api-access-j5tft\") pod \"lws-controller-manager-85985676b5-4gsh6\" (UID: \"7ed304aa-a520-4060-ab6a-f706761b322c\") " pod="openshift-lws-operator/lws-controller-manager-85985676b5-4gsh6"
Sep 18 13:21:36.783567 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:36.783350    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ed304aa-a520-4060-ab6a-f706761b322c-cert\") pod \"lws-controller-manager-85985676b5-4gsh6\" (UID: \"7ed304aa-a520-4060-ab6a-f706761b322c\") " pod="openshift-lws-operator/lws-controller-manager-85985676b5-4gsh6"
Sep 18 13:21:36.783567 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:36.783454    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ed304aa-a520-4060-ab6a-f706761b322c-metrics-cert\") pod \"lws-controller-manager-85985676b5-4gsh6\" (UID: \"7ed304aa-a520-4060-ab6a-f706761b322c\") " pod="openshift-lws-operator/lws-controller-manager-85985676b5-4gsh6"
Sep 18 13:21:36.784084 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:36.784061    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7ed304aa-a520-4060-ab6a-f706761b322c-manager-config\") pod \"lws-controller-manager-85985676b5-4gsh6\" (UID: \"7ed304aa-a520-4060-ab6a-f706761b322c\") " pod="openshift-lws-operator/lws-controller-manager-85985676b5-4gsh6"
Sep 18 13:21:36.786225 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:36.786200    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ed304aa-a520-4060-ab6a-f706761b322c-metrics-cert\") pod \"lws-controller-manager-85985676b5-4gsh6\" (UID: \"7ed304aa-a520-4060-ab6a-f706761b322c\") " pod="openshift-lws-operator/lws-controller-manager-85985676b5-4gsh6"
Sep 18 13:21:36.786829 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:36.786803    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ed304aa-a520-4060-ab6a-f706761b322c-cert\") pod \"lws-controller-manager-85985676b5-4gsh6\" (UID: \"7ed304aa-a520-4060-ab6a-f706761b322c\") " pod="openshift-lws-operator/lws-controller-manager-85985676b5-4gsh6"
Sep 18 13:21:36.805611 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:36.805561    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5tft\" (UniqueName: \"kubernetes.io/projected/7ed304aa-a520-4060-ab6a-f706761b322c-kube-api-access-j5tft\") pod \"lws-controller-manager-85985676b5-4gsh6\" (UID: \"7ed304aa-a520-4060-ab6a-f706761b322c\") " pod="openshift-lws-operator/lws-controller-manager-85985676b5-4gsh6"
Sep 18 13:21:36.971653 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:36.971548    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85985676b5-4gsh6"
Sep 18 13:21:37.268338 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:37.268293    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85985676b5-4gsh6"]
Sep 18 13:21:38.013246 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:21:38.013205    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed304aa_a520_4060_ab6a_f706761b322c.slice/crio-ea5aed1de6c00db8c9a2e62302c41c3e1a46ac6e85ad4a55e9d08de9fc444862 WatchSource:0}: Error finding container ea5aed1de6c00db8c9a2e62302c41c3e1a46ac6e85ad4a55e9d08de9fc444862: Status 404 returned error can't find the container with id ea5aed1de6c00db8c9a2e62302c41c3e1a46ac6e85ad4a55e9d08de9fc444862
Sep 18 13:21:38.048134 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:38.048100    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:21:38.048204 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:38.048169    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:21:38.966015 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:38.965975    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8" event={"ID":"a8c4f346-cfd7-4e0d-b482-2f670e7817d4","Type":"ContainerStarted","Data":"a71eb74cb7e64395c6d3467248256c275326e01c6b1423e2dd09e65587c4b2e3"}
Sep 18 13:21:38.966201 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:38.966075    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:38.968376 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:38.968265    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85985676b5-4gsh6" event={"ID":"7ed304aa-a520-4060-ab6a-f706761b322c","Type":"ContainerStarted","Data":"ea5aed1de6c00db8c9a2e62302c41c3e1a46ac6e85ad4a55e9d08de9fc444862"}
Sep 18 13:21:39.122350 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:39.122284    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8" podStartSLOduration=2.601995561 podStartE2EDuration="5.122259785s" totalImagesPullingTime="2.520264224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:21:39.114272252 +0000 UTC m=+407.549148661" watchObservedRunningTime="2026-09-18 13:21:39.122259785 +0000 UTC m=+407.557136189"
Sep 18 13:21:39.978822 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:39.978788    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-8lgh8"
Sep 18 13:21:40.990667 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:40.990625    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85985676b5-4gsh6" event={"ID":"7ed304aa-a520-4060-ab6a-f706761b322c","Type":"ContainerStarted","Data":"9e1e609d53ae072cca3a53d7f9cf3e6caa19d75647c009d54ccc80bdb6d62081"}
Sep 18 13:21:40.991124 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:40.990908    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-85985676b5-4gsh6"
Sep 18 13:21:41.013015 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:41.012677    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-85985676b5-4gsh6" podStartSLOduration=2.537115079 podStartE2EDuration="5.012658227s" totalImagesPullingTime="2.475543148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:21:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:21:41.012075313 +0000 UTC m=+409.446951720" watchObservedRunningTime="2026-09-18 13:21:41.012658227 +0000 UTC m=+409.447534634"
Sep 18 13:21:43.935085 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:43.935057    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-zc529"
Sep 18 13:21:43.937149 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:43.937128    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-sx4ff"
Sep 18 13:21:51.997219 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:21:51.997181    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-85985676b5-4gsh6"
Sep 18 13:24:51.977428 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:24:51.977400    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k86vn_55e33e14-3f7a-4793-a724-9fd5022740d7/console-operator/1.log"
Sep 18 13:24:51.978230 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:24:51.978211    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k86vn_55e33e14-3f7a-4793-a724-9fd5022740d7/console-operator/1.log"
Sep 18 13:28:48.687257 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.687223    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"]
Sep 18 13:28:48.690926 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.690907    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.694876 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.694856    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-34aeca-openshift-default-dockercfg-v8l5q\""
Sep 18 13:28:48.704499 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.704474    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"]
Sep 18 13:28:48.783964 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.783922    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istiod-ca-cert\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.784155 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.783983    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-credential-socket\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.784155 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.784085    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istio-token\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.784155 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.784143    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istio-podinfo\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.784300 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.784189    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istio-envoy\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.784300 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.784248    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm66l\" (UniqueName: \"kubernetes.io/projected/03f2a203-ca49-4f5c-bcdd-58fc57490d94-kube-api-access-pm66l\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.784370 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.784317    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istio-data\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.784370 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.784355    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-workload-certs\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.784440 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.784391    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-workload-socket\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.855579 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.855546    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"]
Sep 18 13:28:48.859028 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.859010    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:48.861185 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.861164    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-951469-openshift-default-dockercfg-d9qz8\""
Sep 18 13:28:48.872495 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.872464    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"]
Sep 18 13:28:48.885165 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.885138    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istio-data\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.885303 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.885181    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-workload-certs\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.885303 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.885213    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-workload-socket\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.885303 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.885248    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istiod-ca-cert\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.885303 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.885282    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-credential-socket\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.885519 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.885322    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istio-token\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.885519 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.885367    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istio-podinfo\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.885519 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.885409    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istio-envoy\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.885519 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.885453    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm66l\" (UniqueName: \"kubernetes.io/projected/03f2a203-ca49-4f5c-bcdd-58fc57490d94-kube-api-access-pm66l\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.885745 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.885619    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istio-data\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.885745 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.885641    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-workload-certs\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.885850 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.885748    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-workload-socket\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.885850 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.885753    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-credential-socket\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.886026 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.886001    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istiod-ca-cert\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.888100 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.888075    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istio-envoy\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.888234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.888216    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istio-podinfo\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.894438 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.894418    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istio-token\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.897227 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.897203    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm66l\" (UniqueName: \"kubernetes.io/projected/03f2a203-ca49-4f5c-bcdd-58fc57490d94-kube-api-access-pm66l\") pod \"e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") " pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:48.986258 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.986162    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-istio-envoy\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:48.986258 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.986200    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a1a326-b1b7-4266-9907-d729cabc4d58-istio-podinfo\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:48.986258 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.986241    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-workload-socket\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:48.986500 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.986316    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-workload-certs\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:48.986500 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.986378    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-credential-socket\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:48.986500 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.986430    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a1a326-b1b7-4266-9907-d729cabc4d58-istio-token\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:48.986677 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.986519    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a1a326-b1b7-4266-9907-d729cabc4d58-istiod-ca-cert\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:48.986677 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.986583    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-istio-data\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:48.986677 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:48.986634    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgpxx\" (UniqueName: \"kubernetes.io/projected/26a1a326-b1b7-4266-9907-d729cabc4d58-kube-api-access-pgpxx\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:49.001148 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.001116    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:49.087081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.087045    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a1a326-b1b7-4266-9907-d729cabc4d58-istiod-ca-cert\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:49.087249 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.087105    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-istio-data\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:49.087249 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.087131    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgpxx\" (UniqueName: \"kubernetes.io/projected/26a1a326-b1b7-4266-9907-d729cabc4d58-kube-api-access-pgpxx\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:49.087249 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.087159    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-istio-envoy\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:49.087249 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.087180    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a1a326-b1b7-4266-9907-d729cabc4d58-istio-podinfo\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:49.087249 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.087217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-workload-socket\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:49.087408 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.087259    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-workload-certs\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:49.087408 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.087318    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-credential-socket\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:49.087408 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.087342    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a1a326-b1b7-4266-9907-d729cabc4d58-istio-token\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:49.088882 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.088083    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-workload-certs\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:49.088882 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.088164    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-credential-socket\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:49.088882 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.088391    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-workload-socket\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:49.088882 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.088625    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-istio-data\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:49.089322 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.088836    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a1a326-b1b7-4266-9907-d729cabc4d58-istiod-ca-cert\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:49.090857 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.090839    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-istio-envoy\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:49.091497 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.091449    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a1a326-b1b7-4266-9907-d729cabc4d58-istio-podinfo\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:49.096911 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.096857    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgpxx\" (UniqueName: \"kubernetes.io/projected/26a1a326-b1b7-4266-9907-d729cabc4d58-kube-api-access-pgpxx\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:49.097013 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.096941    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a1a326-b1b7-4266-9907-d729cabc4d58-istio-token\") pod \"e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") " pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:49.169088 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.169054    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:49.299802 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.299768    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"]
Sep 18 13:28:49.301013 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:28:49.300984    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f2a203_ca49_4f5c_bcdd_58fc57490d94.slice/crio-61aafcf04b474f3fbb120d57eea4605d5d8672bcc7fd60c5e2afd0fa1a48506c WatchSource:0}: Error finding container 61aafcf04b474f3fbb120d57eea4605d5d8672bcc7fd60c5e2afd0fa1a48506c: Status 404 returned error can't find the container with id 61aafcf04b474f3fbb120d57eea4605d5d8672bcc7fd60c5e2afd0fa1a48506c
Sep 18 13:28:49.303224 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.303191    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:28:49.303318 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.303277    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:28:49.303377 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.303319    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:28:49.457628 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.457585    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"]
Sep 18 13:28:49.460106 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:28:49.460082    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a1a326_b1b7_4266_9907_d729cabc4d58.slice/crio-305b939715df7b25bf0b9933032e627cde016c1fee0f8c1c05c09062aef52719 WatchSource:0}: Error finding container 305b939715df7b25bf0b9933032e627cde016c1fee0f8c1c05c09062aef52719: Status 404 returned error can't find the container with id 305b939715df7b25bf0b9933032e627cde016c1fee0f8c1c05c09062aef52719
Sep 18 13:28:49.462782 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.462749    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:28:49.462889 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.462815    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:28:49.462889 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.462849    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:28:49.991999 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.991955    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt" event={"ID":"03f2a203-ca49-4f5c-bcdd-58fc57490d94","Type":"ContainerStarted","Data":"fb4818768db14ee04b5bc6d801cd5345a07c92a288fec149cf04f7f47c7e090d"}
Sep 18 13:28:49.992454 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.992064    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt" event={"ID":"03f2a203-ca49-4f5c-bcdd-58fc57490d94","Type":"ContainerStarted","Data":"61aafcf04b474f3fbb120d57eea4605d5d8672bcc7fd60c5e2afd0fa1a48506c"}
Sep 18 13:28:49.993778 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.993750    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9" event={"ID":"26a1a326-b1b7-4266-9907-d729cabc4d58","Type":"ContainerStarted","Data":"9def3daef3c488564ab79620d8a2c52e4d0874e7f3a389b59e0c2cc44a66beb0"}
Sep 18 13:28:49.993893 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:49.993797    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9" event={"ID":"26a1a326-b1b7-4266-9907-d729cabc4d58","Type":"ContainerStarted","Data":"305b939715df7b25bf0b9933032e627cde016c1fee0f8c1c05c09062aef52719"}
Sep 18 13:28:50.001375 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:50.001345    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:50.006991 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:50.006964    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:50.035339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:50.035289    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt" podStartSLOduration=2.035274287 podStartE2EDuration="2.035274287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:28:50.033318917 +0000 UTC m=+838.468195321" watchObservedRunningTime="2026-09-18 13:28:50.035274287 +0000 UTC m=+838.470150692"
Sep 18 13:28:50.098375 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:50.098319    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9" podStartSLOduration=2.098301224 podStartE2EDuration="2.098301224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:28:50.096794444 +0000 UTC m=+838.531670852" watchObservedRunningTime="2026-09-18 13:28:50.098301224 +0000 UTC m=+838.533177629"
Sep 18 13:28:50.169286 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:50.169249    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:50.174559 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:50.174537    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:50.999173 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:50.999129    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:50.999658 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:50.999304    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:51.000089 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:51.000066    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:28:51.000201 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:51.000134    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:28:56.774294 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:56.774258    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn"]
Sep 18 13:28:56.779099 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:56.779075    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn"
Sep 18 13:28:56.781982 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:56.781960    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-5be8a3\""
Sep 18 13:28:56.791324 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:56.791295    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn"]
Sep 18 13:28:56.865140 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:56.865101    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fd8a897-c4ff-4183-8ab5-412d5a406847-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn\" (UID: \"7fd8a897-c4ff-4183-8ab5-412d5a406847\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn"
Sep 18 13:28:56.865140 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:56.865139    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vkdw\" (UniqueName: \"kubernetes.io/projected/7fd8a897-c4ff-4183-8ab5-412d5a406847-kube-api-access-9vkdw\") pod \"payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn\" (UID: \"7fd8a897-c4ff-4183-8ab5-412d5a406847\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn"
Sep 18 13:28:56.966698 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:56.966657    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fd8a897-c4ff-4183-8ab5-412d5a406847-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn\" (UID: \"7fd8a897-c4ff-4183-8ab5-412d5a406847\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn"
Sep 18 13:28:56.966698 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:56.966700    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vkdw\" (UniqueName: \"kubernetes.io/projected/7fd8a897-c4ff-4183-8ab5-412d5a406847-kube-api-access-9vkdw\") pod \"payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn\" (UID: \"7fd8a897-c4ff-4183-8ab5-412d5a406847\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn"
Sep 18 13:28:56.967449 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:56.967419    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fd8a897-c4ff-4183-8ab5-412d5a406847-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn\" (UID: \"7fd8a897-c4ff-4183-8ab5-412d5a406847\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn"
Sep 18 13:28:56.975141 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:56.975110    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vkdw\" (UniqueName: \"kubernetes.io/projected/7fd8a897-c4ff-4183-8ab5-412d5a406847-kube-api-access-9vkdw\") pod \"payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn\" (UID: \"7fd8a897-c4ff-4183-8ab5-412d5a406847\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn"
Sep 18 13:28:57.090273 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:57.090185    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn"
Sep 18 13:28:57.364633 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:57.364589    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn"]
Sep 18 13:28:57.367068 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:28:57.367031    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd8a897_c4ff_4183_8ab5_412d5a406847.slice/crio-8f10457b627c191ea2db32f0d3c1f17cbc06d716b2de7590647833f7977a247e WatchSource:0}: Error finding container 8f10457b627c191ea2db32f0d3c1f17cbc06d716b2de7590647833f7977a247e: Status 404 returned error can't find the container with id 8f10457b627c191ea2db32f0d3c1f17cbc06d716b2de7590647833f7977a247e
Sep 18 13:28:57.368912 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:57.368895    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 13:28:58.033271 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.033230    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn" event={"ID":"7fd8a897-c4ff-4183-8ab5-412d5a406847","Type":"ContainerStarted","Data":"8f10457b627c191ea2db32f0d3c1f17cbc06d716b2de7590647833f7977a247e"}
Sep 18 13:28:58.171521 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.171470    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"]
Sep 18 13:28:58.175352 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.175323    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.177464 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.177443    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a304aa20-openshift-default-dockercfg-wtxqx\""
Sep 18 13:28:58.187785 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.187756    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"]
Sep 18 13:28:58.277135 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.277093    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-credential-socket\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.277135 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.277148    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32c7f112-cb93-405d-baab-8b0d9c28c54f-istio-token\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.277399 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.277218    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32c7f112-cb93-405d-baab-8b0d9c28c54f-istio-podinfo\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.277399 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.277306    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-istio-data\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.277399 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.277335    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32c7f112-cb93-405d-baab-8b0d9c28c54f-istiod-ca-cert\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.277399 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.277388    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-workload-socket\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.277640 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.277425    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tk7j\" (UniqueName: \"kubernetes.io/projected/32c7f112-cb93-405d-baab-8b0d9c28c54f-kube-api-access-5tk7j\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.277640 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.277461    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-workload-certs\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.277640 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.277523    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-istio-envoy\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.378198 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.378144    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-credential-socket\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.378383 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.378210    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32c7f112-cb93-405d-baab-8b0d9c28c54f-istio-token\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.378383 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.378245    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32c7f112-cb93-405d-baab-8b0d9c28c54f-istio-podinfo\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.378383 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.378303    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-istio-data\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.378383 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.378332    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32c7f112-cb93-405d-baab-8b0d9c28c54f-istiod-ca-cert\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.378383 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.378377    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-workload-socket\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.378692 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.378506    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tk7j\" (UniqueName: \"kubernetes.io/projected/32c7f112-cb93-405d-baab-8b0d9c28c54f-kube-api-access-5tk7j\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.378692 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.378566    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-workload-certs\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.378692 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.378661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-istio-envoy\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.378853 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.378709    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-credential-socket\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.378853 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.378740    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-istio-data\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.378853 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.378774    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-workload-socket\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.379004 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.378942    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-workload-certs\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.379252 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.379215    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32c7f112-cb93-405d-baab-8b0d9c28c54f-istiod-ca-cert\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.381121 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.381093    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-istio-envoy\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.381308 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.381286    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32c7f112-cb93-405d-baab-8b0d9c28c54f-istio-podinfo\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.389447 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.389426    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32c7f112-cb93-405d-baab-8b0d9c28c54f-istio-token\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.389962 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.389935    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tk7j\" (UniqueName: \"kubernetes.io/projected/32c7f112-cb93-405d-baab-8b0d9c28c54f-kube-api-access-5tk7j\") pod \"e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") " pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.488640 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.488609    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:58.797528 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.797503    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"]
Sep 18 13:28:58.799095 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:28:58.799070    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c7f112_cb93_405d_baab_8b0d9c28c54f.slice/crio-5461583c981077779785ac0a57b29f59c95cf43bb05a8df249a104adb91575dd WatchSource:0}: Error finding container 5461583c981077779785ac0a57b29f59c95cf43bb05a8df249a104adb91575dd: Status 404 returned error can't find the container with id 5461583c981077779785ac0a57b29f59c95cf43bb05a8df249a104adb91575dd
Sep 18 13:28:58.801311 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.801281    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:28:58.801381 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.801344    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:28:58.801381 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:58.801373    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:28:59.041082 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:59.041048    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv" event={"ID":"32c7f112-cb93-405d-baab-8b0d9c28c54f","Type":"ContainerStarted","Data":"45db1f9749b53cfadc53417ded5de1aedeb00402013f7a0c77de7d9f327ffe38"}
Sep 18 13:28:59.041705 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:59.041481    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv" event={"ID":"32c7f112-cb93-405d-baab-8b0d9c28c54f","Type":"ContainerStarted","Data":"5461583c981077779785ac0a57b29f59c95cf43bb05a8df249a104adb91575dd"}
Sep 18 13:28:59.043107 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:59.043076    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn" event={"ID":"7fd8a897-c4ff-4183-8ab5-412d5a406847","Type":"ContainerStarted","Data":"cdd158261e83ae82e497885f7e6a672f1b0243e8193f119493fd394948c56b64"}
Sep 18 13:28:59.043219 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:59.043134    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn"
Sep 18 13:28:59.067781 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:59.066656    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv" podStartSLOduration=1.066636473 podStartE2EDuration="1.066636473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:28:59.064187188 +0000 UTC m=+847.499063595" watchObservedRunningTime="2026-09-18 13:28:59.066636473 +0000 UTC m=+847.501512875"
Sep 18 13:28:59.087670 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:59.087587    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn" podStartSLOduration=1.987253708 podStartE2EDuration="3.087570864s" totalImagesPullingTime="1.100317156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:28:59.084916386 +0000 UTC m=+847.519792791" watchObservedRunningTime="2026-09-18 13:28:59.087570864 +0000 UTC m=+847.522447268"
Sep 18 13:28:59.489342 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:59.489300    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:28:59.494756 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:28:59.494723    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:29:00.048292 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:00.048256    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:29:00.049140 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:00.049117    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:29:01.160259 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.160229    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"]
Sep 18 13:29:01.163553 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.163532    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.167539 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.167514    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-b7292e-openshift-default-dockercfg-qxsq5\""
Sep 18 13:29:01.245665 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.245629    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"]
Sep 18 13:29:01.307828 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.307790    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istiod-ca-cert\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.308012 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.307842    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-credential-socket\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.308012 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.307921    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istio-envoy\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.308012 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.307996    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-workload-socket\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.308152 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.308029    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istio-data\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.308152 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.308113    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-workload-certs\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.308152 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.308141    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkt29\" (UniqueName: \"kubernetes.io/projected/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-kube-api-access-mkt29\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.308263 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.308167    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istio-token\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.308297 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.308283    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istio-podinfo\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.409404 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.409362    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-credential-socket\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.409639 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.409411    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istio-envoy\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.409639 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.409459    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-workload-socket\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.409639 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.409484    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istio-data\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.409639 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.409537    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-workload-certs\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.409639 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.409565    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkt29\" (UniqueName: \"kubernetes.io/projected/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-kube-api-access-mkt29\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.409639 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.409587    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istio-token\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.409971 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.409811    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-credential-socket\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.409971 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.409839    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istio-podinfo\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.409971 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.409862    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-workload-socket\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.409971 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.409928    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istiod-ca-cert\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.410136 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.410010    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-workload-certs\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.410177 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.410158    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istio-data\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.410563 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.410502    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istiod-ca-cert\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.411676 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.411653    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istio-envoy\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.412247 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.412232    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istio-podinfo\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.430810 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.430782    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkt29\" (UniqueName: \"kubernetes.io/projected/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-kube-api-access-mkt29\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.437066 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.437037    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istio-token\") pod \"e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") " pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.476917 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.476882    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:01.807063 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.806987    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"]
Sep 18 13:29:01.824816 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:29:01.824752    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dca7ee1_59ae_440b_b6fa_b1dc1d32be20.slice/crio-da84f9253bb3257148b6b63cb105e29945aac7428e2301d659d5d0812aea1433 WatchSource:0}: Error finding container da84f9253bb3257148b6b63cb105e29945aac7428e2301d659d5d0812aea1433: Status 404 returned error can't find the container with id da84f9253bb3257148b6b63cb105e29945aac7428e2301d659d5d0812aea1433
Sep 18 13:29:01.827895 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.827848    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:29:01.827991 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.827968    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:29:01.828055 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:01.828013    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:29:02.063673 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:02.063558    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc" event={"ID":"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20","Type":"ContainerStarted","Data":"6093db13ce1ea90b7ea71f2a0b912624c551d7ddfe48d1ba9724c86a46654173"}
Sep 18 13:29:02.063673 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:02.063637    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc" event={"ID":"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20","Type":"ContainerStarted","Data":"da84f9253bb3257148b6b63cb105e29945aac7428e2301d659d5d0812aea1433"}
Sep 18 13:29:02.094897 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:02.094833    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc" podStartSLOduration=1.094809238 podStartE2EDuration="1.094809238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:29:02.092369461 +0000 UTC m=+850.527245867" watchObservedRunningTime="2026-09-18 13:29:02.094809238 +0000 UTC m=+850.529685643"
Sep 18 13:29:02.478064 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:02.478028    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:02.482900 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:02.482873    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:03.072521 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:03.072479    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:03.078636 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:03.078580    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:07.772630 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.771537    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"]
Sep 18 13:29:07.776189 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.776165    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.783393 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.783368    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-021eb8-openshift-default-dockercfg-dtl22\""
Sep 18 13:29:07.801859 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.801830    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"]
Sep 18 13:29:07.894251 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.893089    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03837341-329a-4b27-827e-4b95a242d96e-istio-token\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.894251 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.893152    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03837341-329a-4b27-827e-4b95a242d96e-istiod-ca-cert\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.894251 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.893195    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-workload-certs\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.894251 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.893254    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-workload-socket\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.894251 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.893333    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-istio-envoy\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.894251 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.893366    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03837341-329a-4b27-827e-4b95a242d96e-istio-podinfo\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.894251 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.893396    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97lwc\" (UniqueName: \"kubernetes.io/projected/03837341-329a-4b27-827e-4b95a242d96e-kube-api-access-97lwc\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.894251 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.893429    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-credential-socket\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.894251 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.893461    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-istio-data\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.994478 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.994436    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-workload-socket\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.994719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.994531    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-istio-envoy\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.994719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.994562    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03837341-329a-4b27-827e-4b95a242d96e-istio-podinfo\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.994719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.994590    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97lwc\" (UniqueName: \"kubernetes.io/projected/03837341-329a-4b27-827e-4b95a242d96e-kube-api-access-97lwc\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.994719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.994635    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-credential-socket\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.994719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.994666    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-istio-data\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.995203 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.994724    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03837341-329a-4b27-827e-4b95a242d96e-istio-token\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.995203 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.994747    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03837341-329a-4b27-827e-4b95a242d96e-istiod-ca-cert\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.995203 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.994789    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-workload-certs\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.995203 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.994954    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-workload-socket\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.995405 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.995305    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-istio-data\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.995457 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.995403    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-workload-certs\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.995572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.995500    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03837341-329a-4b27-827e-4b95a242d96e-istiod-ca-cert\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.995676 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.995646    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-credential-socket\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.997211 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.997187    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-istio-envoy\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:07.997360 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:07.997236    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03837341-329a-4b27-827e-4b95a242d96e-istio-podinfo\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:08.035987 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:08.035910    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97lwc\" (UniqueName: \"kubernetes.io/projected/03837341-329a-4b27-827e-4b95a242d96e-kube-api-access-97lwc\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:08.036223 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:08.036076    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03837341-329a-4b27-827e-4b95a242d96e-istio-token\") pod \"e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") " pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:08.101524 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:08.101495    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:08.392296 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:08.392270    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"]
Sep 18 13:29:08.395075 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:29:08.395041    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03837341_329a_4b27_827e_4b95a242d96e.slice/crio-5637289a147081c450c0333a9f47aec6afd6dd4e468501d77082137a6ca36a9f WatchSource:0}: Error finding container 5637289a147081c450c0333a9f47aec6afd6dd4e468501d77082137a6ca36a9f: Status 404 returned error can't find the container with id 5637289a147081c450c0333a9f47aec6afd6dd4e468501d77082137a6ca36a9f
Sep 18 13:29:08.397283 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:08.397249    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:29:08.397374 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:08.397321    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:29:08.397374 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:08.397348    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:29:09.127074 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:09.127032    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb" event={"ID":"03837341-329a-4b27-827e-4b95a242d96e","Type":"ContainerStarted","Data":"f3610132459dba62fbe9c245d6b469865045dc2edaf1042d83e49417b3424ac7"}
Sep 18 13:29:09.127520 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:09.127112    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb" event={"ID":"03837341-329a-4b27-827e-4b95a242d96e","Type":"ContainerStarted","Data":"5637289a147081c450c0333a9f47aec6afd6dd4e468501d77082137a6ca36a9f"}
Sep 18 13:29:09.162231 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:09.162175    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb" podStartSLOduration=2.16215636 podStartE2EDuration="2.16215636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:29:09.158261056 +0000 UTC m=+857.593137460" watchObservedRunningTime="2026-09-18 13:29:09.16215636 +0000 UTC m=+857.597032763"
Sep 18 13:29:10.050579 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:10.050546    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn"
Sep 18 13:29:10.102645 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:10.102589    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:10.109499 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:10.109465    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:10.134453 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:10.134411    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:10.135547 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:10.135514    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:12.146565 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:12.146530    2566 generic.go:356] "Generic (PLEG): container finished" podID="03f2a203-ca49-4f5c-bcdd-58fc57490d94" containerID="fb4818768db14ee04b5bc6d801cd5345a07c92a288fec149cf04f7f47c7e090d" exitCode=0
Sep 18 13:29:12.147096 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:12.146626    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt" event={"ID":"03f2a203-ca49-4f5c-bcdd-58fc57490d94","Type":"ContainerDied","Data":"fb4818768db14ee04b5bc6d801cd5345a07c92a288fec149cf04f7f47c7e090d"}
Sep 18 13:29:12.147096 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:12.147078    2566 scope.go:117] "RemoveContainer" containerID="fb4818768db14ee04b5bc6d801cd5345a07c92a288fec149cf04f7f47c7e090d"
Sep 18 13:29:12.148458 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:12.148427    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:29:12.148540 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:12.148498    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:29:12.148540 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:12.148503    2566 generic.go:356] "Generic (PLEG): container finished" podID="26a1a326-b1b7-4266-9907-d729cabc4d58" containerID="9def3daef3c488564ab79620d8a2c52e4d0874e7f3a389b59e0c2cc44a66beb0" exitCode=0
Sep 18 13:29:12.148629 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:12.148556    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:29:12.148629 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:12.148579    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9" event={"ID":"26a1a326-b1b7-4266-9907-d729cabc4d58","Type":"ContainerDied","Data":"9def3daef3c488564ab79620d8a2c52e4d0874e7f3a389b59e0c2cc44a66beb0"}
Sep 18 13:29:12.149105 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:12.149056    2566 scope.go:117] "RemoveContainer" containerID="9def3daef3c488564ab79620d8a2c52e4d0874e7f3a389b59e0c2cc44a66beb0"
Sep 18 13:29:12.150111 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:12.150081    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:29:12.150181 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:12.150156    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:29:12.150220 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:12.150201    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:29:13.164089 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:13.164038    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9" event={"ID":"26a1a326-b1b7-4266-9907-d729cabc4d58","Type":"ContainerStarted","Data":"dc6ac75ebbd7b1f5f079deb52058e87498e767d6f3a2a5dfd9387e760d304475"}
Sep 18 13:29:13.168640 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:13.168585    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt" event={"ID":"03f2a203-ca49-4f5c-bcdd-58fc57490d94","Type":"ContainerStarted","Data":"f684fe0c7916f50534650b5eb38ed5a70cf469f8150b2635eb91f439b981e87b"}
Sep 18 13:29:13.169684 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:13.169656    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:29:13.169807 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:13.169761    2566 patch_prober.go:28] interesting pod/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 18 13:29:13.169871 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:13.169804    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9" podUID="26a1a326-b1b7-4266-9907-d729cabc4d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 18 13:29:14.001790 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:14.001746    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:29:14.002144 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:14.002120    2566 patch_prober.go:28] interesting pod/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 18 13:29:14.002398 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:14.002169    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt" podUID="03f2a203-ca49-4f5c-bcdd-58fc57490d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 18 13:29:14.171164 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:14.171121    2566 patch_prober.go:28] interesting pod/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 18 13:29:14.171680 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:14.171192    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9" podUID="26a1a326-b1b7-4266-9907-d729cabc4d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 18 13:29:15.001741 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:15.001703    2566 patch_prober.go:28] interesting pod/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 18 13:29:15.001926 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:15.001760    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt" podUID="03f2a203-ca49-4f5c-bcdd-58fc57490d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 18 13:29:15.170134 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:15.170087    2566 patch_prober.go:28] interesting pod/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 18 13:29:15.170349 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:15.170157    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9" podUID="26a1a326-b1b7-4266-9907-d729cabc4d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 18 13:29:16.001668 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:16.001627    2566 patch_prober.go:28] interesting pod/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 18 13:29:16.002076 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:16.001703    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt" podUID="03f2a203-ca49-4f5c-bcdd-58fc57490d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 18 13:29:16.169814 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:16.169764    2566 patch_prober.go:28] interesting pod/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 18 13:29:16.170038 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:16.169842    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9" podUID="26a1a326-b1b7-4266-9907-d729cabc4d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 18 13:29:17.002633 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:17.002557    2566 patch_prober.go:28] interesting pod/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 18 13:29:17.003165 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:17.002671    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt" podUID="03f2a203-ca49-4f5c-bcdd-58fc57490d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 18 13:29:17.176622 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:17.176007    2566 patch_prober.go:28] interesting pod/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 18 13:29:17.176622 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:17.176072    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9" podUID="26a1a326-b1b7-4266-9907-d729cabc4d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 18 13:29:18.002343 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:18.002289    2566 patch_prober.go:28] interesting pod/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Sep 18 13:29:18.002572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:18.002366    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt" podUID="03f2a203-ca49-4f5c-bcdd-58fc57490d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Sep 18 13:29:18.174621 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:18.173004    2566 patch_prober.go:28] interesting pod/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 18 13:29:18.174621 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:18.173109    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9" podUID="26a1a326-b1b7-4266-9907-d729cabc4d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 18 13:29:19.006615 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:19.006565    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:29:19.006877 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:19.006860    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:29:19.007687 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:19.007669    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:29:19.187413 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:19.185907    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:29:19.187413 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:19.187249    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:29:19.188931 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:19.188913    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:29:21.117421 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:21.117384    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a304aa20-fcfcm"]
Sep 18 13:29:21.120142 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:21.120125    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a304aa20-fcfcm"
Sep 18 13:29:21.125490 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:21.125470    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:29:21.127604 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:21.127568    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nh2cd\""
Sep 18 13:29:21.141122 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:21.141088    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a304aa20-fcfcm"]
Sep 18 13:29:21.189456 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:21.189417    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b86881-41ca-4cb0-a08d-e66cdb9f434e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a304aa20-fcfcm\" (UID: \"82b86881-41ca-4cb0-a08d-e66cdb9f434e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a304aa20-fcfcm"
Sep 18 13:29:21.291099 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:21.291040    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b86881-41ca-4cb0-a08d-e66cdb9f434e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a304aa20-fcfcm\" (UID: \"82b86881-41ca-4cb0-a08d-e66cdb9f434e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a304aa20-fcfcm"
Sep 18 13:29:21.292001 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:21.291968    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b86881-41ca-4cb0-a08d-e66cdb9f434e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a304aa20-fcfcm\" (UID: \"82b86881-41ca-4cb0-a08d-e66cdb9f434e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a304aa20-fcfcm"
Sep 18 13:29:21.430635 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:21.430533    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a304aa20-fcfcm"
Sep 18 13:29:21.762188 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:21.762027    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a304aa20-fcfcm"]
Sep 18 13:29:22.235132 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:22.235104    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a304aa20-fcfcm" event={"ID":"82b86881-41ca-4cb0-a08d-e66cdb9f434e","Type":"ContainerStarted","Data":"5e6fca4808ee09d53dabdd1cc13473508c0a64c1ee585174c7d11f8fa059f3dd"}
Sep 18 13:29:25.259835 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:25.259780    2566 generic.go:356] "Generic (PLEG): container finished" podID="82b86881-41ca-4cb0-a08d-e66cdb9f434e" containerID="12b0de2158e424c88d5da600c27221e7ddd0ecf4cb754472ed7b79cf4dcf141c" exitCode=0
Sep 18 13:29:25.260320 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:25.259852    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a304aa20-fcfcm" event={"ID":"82b86881-41ca-4cb0-a08d-e66cdb9f434e","Type":"ContainerDied","Data":"12b0de2158e424c88d5da600c27221e7ddd0ecf4cb754472ed7b79cf4dcf141c"}
Sep 18 13:29:26.266998 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:26.266909    2566 generic.go:356] "Generic (PLEG): container finished" podID="32c7f112-cb93-405d-baab-8b0d9c28c54f" containerID="45db1f9749b53cfadc53417ded5de1aedeb00402013f7a0c77de7d9f327ffe38" exitCode=0
Sep 18 13:29:26.266998 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:26.266979    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv" event={"ID":"32c7f112-cb93-405d-baab-8b0d9c28c54f","Type":"ContainerDied","Data":"45db1f9749b53cfadc53417ded5de1aedeb00402013f7a0c77de7d9f327ffe38"}
Sep 18 13:29:26.267520 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:26.267408    2566 scope.go:117] "RemoveContainer" containerID="45db1f9749b53cfadc53417ded5de1aedeb00402013f7a0c77de7d9f327ffe38"
Sep 18 13:29:26.268694 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:26.268674    2566 generic.go:356] "Generic (PLEG): container finished" podID="1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" containerID="6093db13ce1ea90b7ea71f2a0b912624c551d7ddfe48d1ba9724c86a46654173" exitCode=0
Sep 18 13:29:26.268694 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:26.268676    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:29:26.268804 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:26.268743    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:29:26.268804 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:26.268780    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:29:26.268871 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:26.268746    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc" event={"ID":"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20","Type":"ContainerDied","Data":"6093db13ce1ea90b7ea71f2a0b912624c551d7ddfe48d1ba9724c86a46654173"}
Sep 18 13:29:26.269258 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:26.269239    2566 scope.go:117] "RemoveContainer" containerID="6093db13ce1ea90b7ea71f2a0b912624c551d7ddfe48d1ba9724c86a46654173"
Sep 18 13:29:26.270405 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:26.270376    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:29:26.270501 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:26.270442    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:29:26.270501 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:26.270488    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:29:26.451058 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:26.451026    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a304aa20-fcfcm"
Sep 18 13:29:26.550359 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:26.550267    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b86881-41ca-4cb0-a08d-e66cdb9f434e-maas-api-service-ca\") pod \"82b86881-41ca-4cb0-a08d-e66cdb9f434e\" (UID: \"82b86881-41ca-4cb0-a08d-e66cdb9f434e\") "
Sep 18 13:29:26.550795 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:26.550764    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82b86881-41ca-4cb0-a08d-e66cdb9f434e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "82b86881-41ca-4cb0-a08d-e66cdb9f434e" (UID: "82b86881-41ca-4cb0-a08d-e66cdb9f434e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:29:26.551082 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:26.551061    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82b86881-41ca-4cb0-a08d-e66cdb9f434e-maas-api-service-ca\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:29:27.283400 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:27.283343    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a304aa20-fcfcm" event={"ID":"82b86881-41ca-4cb0-a08d-e66cdb9f434e","Type":"ContainerDied","Data":"5e6fca4808ee09d53dabdd1cc13473508c0a64c1ee585174c7d11f8fa059f3dd"}
Sep 18 13:29:27.283918 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:27.283434    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e6fca4808ee09d53dabdd1cc13473508c0a64c1ee585174c7d11f8fa059f3dd"
Sep 18 13:29:27.283918 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:27.283513    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a304aa20-fcfcm"
Sep 18 13:29:27.290907 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:27.290876    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv" event={"ID":"32c7f112-cb93-405d-baab-8b0d9c28c54f","Type":"ContainerStarted","Data":"ba476cc77685193cf166d7a7d8c199d1d31a1e9284d6a4867ac62f8aa7c59599"}
Sep 18 13:29:27.293338 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:27.293244    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc" event={"ID":"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20","Type":"ContainerStarted","Data":"d87c83bdb08e2090753e89efac5e6ad0ce01de2b6b6f3495e3d32191f389bb15"}
Sep 18 13:29:27.477649 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:27.477585    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:27.478099 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:27.477989    2566 patch_prober.go:28] interesting pod/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 18 13:29:27.478277 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:27.478243    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc" podUID="1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 18 13:29:27.489551 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:27.489522    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:29:27.490500 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:27.490471    2566 patch_prober.go:28] interesting pod/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 18 13:29:27.490590 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:27.490538    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv" podUID="32c7f112-cb93-405d-baab-8b0d9c28c54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 18 13:29:28.478159 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:28.478115    2566 patch_prober.go:28] interesting pod/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 18 13:29:28.478786 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:28.478748    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc" podUID="1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 18 13:29:28.489270 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:28.489234    2566 patch_prober.go:28] interesting pod/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 18 13:29:28.489535 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:28.489505    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv" podUID="32c7f112-cb93-405d-baab-8b0d9c28c54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 18 13:29:29.478136 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:29.478083    2566 patch_prober.go:28] interesting pod/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 18 13:29:29.478372 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:29.478158    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc" podUID="1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 18 13:29:29.489425 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:29.489380    2566 patch_prober.go:28] interesting pod/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 18 13:29:29.489627 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:29.489449    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv" podUID="32c7f112-cb93-405d-baab-8b0d9c28c54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 18 13:29:30.048687 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:30.048651    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:29:30.478466 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:30.478417    2566 patch_prober.go:28] interesting pod/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 18 13:29:30.478999 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:30.478498    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc" podUID="1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 18 13:29:30.490192 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:30.490150    2566 patch_prober.go:28] interesting pod/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 18 13:29:30.490380 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:30.490217    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv" podUID="32c7f112-cb93-405d-baab-8b0d9c28c54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 18 13:29:31.478017 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:31.477977    2566 patch_prober.go:28] interesting pod/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 18 13:29:31.478303 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:31.478269    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc" podUID="1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 18 13:29:31.493944 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:31.493901    2566 patch_prober.go:28] interesting pod/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 18 13:29:31.494524 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:31.494488    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv" podUID="32c7f112-cb93-405d-baab-8b0d9c28c54f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 18 13:29:31.820239 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:31.820147    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a304aa20-fcfcm"]
Sep 18 13:29:31.827371 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:31.827336    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a304aa20-fcfcm"]
Sep 18 13:29:32.055619 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:32.053434    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"]
Sep 18 13:29:32.059630 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:32.057900    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv" podUID="32c7f112-cb93-405d-baab-8b0d9c28c54f" containerName="istio-proxy" containerID="cri-o://ba476cc77685193cf166d7a7d8c199d1d31a1e9284d6a4867ac62f8aa7c59599" gracePeriod=30
Sep 18 13:29:32.110536 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:32.110498    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82b86881-41ca-4cb0-a08d-e66cdb9f434e" path="/var/lib/kubelet/pods/82b86881-41ca-4cb0-a08d-e66cdb9f434e/volumes"
Sep 18 13:29:32.477535 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:32.477442    2566 patch_prober.go:28] interesting pod/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 18 13:29:32.477830 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:32.477792    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc" podUID="1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 18 13:29:33.073179 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.073145    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:33.167903 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.160426    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"]
Sep 18 13:29:33.168590 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.168558    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82b86881-41ca-4cb0-a08d-e66cdb9f434e" containerName="revoke-keys"
Sep 18 13:29:33.168792 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.168775    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="82b86881-41ca-4cb0-a08d-e66cdb9f434e" containerName="revoke-keys"
Sep 18 13:29:33.169155 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.169141    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="82b86881-41ca-4cb0-a08d-e66cdb9f434e" containerName="revoke-keys"
Sep 18 13:29:33.177843 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.177816    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"]
Sep 18 13:29:33.178160 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.178102    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.180649 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.180622    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-8178145c-gw-openshift-default-dockercfg-m5hd8\""
Sep 18 13:29:33.250617 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.250289    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f42433c-819f-40ef-a520-df3a5e888160-istiod-ca-cert\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.250617 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.250401    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-istio-envoy\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.250617 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.250429    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-istio-data\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.250617 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.250488    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f42433c-819f-40ef-a520-df3a5e888160-istio-podinfo\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.250617 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.250521    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-credential-socket\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.250617 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.250557    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-workload-certs\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.251079 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.250623    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-workload-socket\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.251079 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.250662    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpj6b\" (UniqueName: \"kubernetes.io/projected/6f42433c-819f-40ef-a520-df3a5e888160-kube-api-access-dpj6b\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.251079 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.250712    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f42433c-819f-40ef-a520-df3a5e888160-istio-token\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.351764 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.351709    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f42433c-819f-40ef-a520-df3a5e888160-istio-podinfo\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.351963 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.351774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-credential-socket\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.351963 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.351813    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-workload-certs\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.351963 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.351862    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-workload-socket\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.351963 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.351887    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpj6b\" (UniqueName: \"kubernetes.io/projected/6f42433c-819f-40ef-a520-df3a5e888160-kube-api-access-dpj6b\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.351963 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.351951    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f42433c-819f-40ef-a520-df3a5e888160-istio-token\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.352509 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.352458    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-workload-socket\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.352509 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.352435    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-workload-certs\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.352724 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.352582    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f42433c-819f-40ef-a520-df3a5e888160-istiod-ca-cert\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.352724 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.352676    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-istio-envoy\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.352724 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.352701    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-istio-data\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.352879 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.352751    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-credential-socket\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.353414 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.353391    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-istio-data\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.353616 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.353565    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f42433c-819f-40ef-a520-df3a5e888160-istiod-ca-cert\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.355136 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.355077    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f42433c-819f-40ef-a520-df3a5e888160-istio-podinfo\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.355420 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.355394    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-istio-envoy\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.366065 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.366017    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f42433c-819f-40ef-a520-df3a5e888160-istio-token\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.371332 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.371288    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpj6b\" (UniqueName: \"kubernetes.io/projected/6f42433c-819f-40ef-a520-df3a5e888160-kube-api-access-dpj6b\") pod \"e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") " pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.481775 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.481745    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:33.482820 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.482801    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:29:33.493484 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.493458    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:33.779956 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.779931    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"]
Sep 18 13:29:33.782559 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:29:33.782532    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f42433c_819f_40ef_a520_df3a5e888160.slice/crio-fa51c43bd6ba1f4ce3064b150ce847973c4d795e2f8959718c34eb5da73d3d80 WatchSource:0}: Error finding container fa51c43bd6ba1f4ce3064b150ce847973c4d795e2f8959718c34eb5da73d3d80: Status 404 returned error can't find the container with id fa51c43bd6ba1f4ce3064b150ce847973c4d795e2f8959718c34eb5da73d3d80
Sep 18 13:29:33.785079 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.785049    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:29:33.785165 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.785112    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:29:33.785165 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:33.785140    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:29:34.350607 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:34.350553    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz" event={"ID":"6f42433c-819f-40ef-a520-df3a5e888160","Type":"ContainerStarted","Data":"88cd44a4b0b08698a9a020c4a90e7dd57a020538c4faea5c404d88f6d258b048"}
Sep 18 13:29:34.351073 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:34.350631    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz" event={"ID":"6f42433c-819f-40ef-a520-df3a5e888160","Type":"ContainerStarted","Data":"fa51c43bd6ba1f4ce3064b150ce847973c4d795e2f8959718c34eb5da73d3d80"}
Sep 18 13:29:34.408567 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:34.408505    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz" podStartSLOduration=1.408479835 podStartE2EDuration="1.408479835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:29:34.403138289 +0000 UTC m=+882.838014692" watchObservedRunningTime="2026-09-18 13:29:34.408479835 +0000 UTC m=+882.843356318"
Sep 18 13:29:34.493703 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:34.493664    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:34.499536 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:34.499504    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:35.355811 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:35.355777    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:35.356905 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:35.356885    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:29:37.368151 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:37.368118    2566 generic.go:356] "Generic (PLEG): container finished" podID="03837341-329a-4b27-827e-4b95a242d96e" containerID="f3610132459dba62fbe9c245d6b469865045dc2edaf1042d83e49417b3424ac7" exitCode=0
Sep 18 13:29:37.368582 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:37.368194    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb" event={"ID":"03837341-329a-4b27-827e-4b95a242d96e","Type":"ContainerDied","Data":"f3610132459dba62fbe9c245d6b469865045dc2edaf1042d83e49417b3424ac7"}
Sep 18 13:29:37.368799 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:37.368780    2566 scope.go:117] "RemoveContainer" containerID="f3610132459dba62fbe9c245d6b469865045dc2edaf1042d83e49417b3424ac7"
Sep 18 13:29:37.369909 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:37.369880    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:29:37.369984 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:37.369957    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:29:37.370024 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:37.369992    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:29:38.322929 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.322905    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:29:38.376414 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.376318    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb" event={"ID":"03837341-329a-4b27-827e-4b95a242d96e","Type":"ContainerStarted","Data":"f0d807229cdc7242e336f3899ce9a9b65903217702c38d5983ea307b9b991fb5"}
Sep 18 13:29:38.378344 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.378317    2566 generic.go:356] "Generic (PLEG): container finished" podID="32c7f112-cb93-405d-baab-8b0d9c28c54f" containerID="ba476cc77685193cf166d7a7d8c199d1d31a1e9284d6a4867ac62f8aa7c59599" exitCode=0
Sep 18 13:29:38.378468 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.378354    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv" event={"ID":"32c7f112-cb93-405d-baab-8b0d9c28c54f","Type":"ContainerDied","Data":"ba476cc77685193cf166d7a7d8c199d1d31a1e9284d6a4867ac62f8aa7c59599"}
Sep 18 13:29:38.378468 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.378379    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"
Sep 18 13:29:38.378468 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.378402    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv" event={"ID":"32c7f112-cb93-405d-baab-8b0d9c28c54f","Type":"ContainerDied","Data":"5461583c981077779785ac0a57b29f59c95cf43bb05a8df249a104adb91575dd"}
Sep 18 13:29:38.378468 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.378439    2566 scope.go:117] "RemoveContainer" containerID="ba476cc77685193cf166d7a7d8c199d1d31a1e9284d6a4867ac62f8aa7c59599"
Sep 18 13:29:38.391787 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.391769    2566 scope.go:117] "RemoveContainer" containerID="45db1f9749b53cfadc53417ded5de1aedeb00402013f7a0c77de7d9f327ffe38"
Sep 18 13:29:38.401933 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.401895    2566 scope.go:117] "RemoveContainer" containerID="ba476cc77685193cf166d7a7d8c199d1d31a1e9284d6a4867ac62f8aa7c59599"
Sep 18 13:29:38.402456 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:29:38.402320    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba476cc77685193cf166d7a7d8c199d1d31a1e9284d6a4867ac62f8aa7c59599\": container with ID starting with ba476cc77685193cf166d7a7d8c199d1d31a1e9284d6a4867ac62f8aa7c59599 not found: ID does not exist" containerID="ba476cc77685193cf166d7a7d8c199d1d31a1e9284d6a4867ac62f8aa7c59599"
Sep 18 13:29:38.402456 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.402361    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba476cc77685193cf166d7a7d8c199d1d31a1e9284d6a4867ac62f8aa7c59599"} err="failed to get container status \"ba476cc77685193cf166d7a7d8c199d1d31a1e9284d6a4867ac62f8aa7c59599\": rpc error: code = NotFound desc = could not find container \"ba476cc77685193cf166d7a7d8c199d1d31a1e9284d6a4867ac62f8aa7c59599\": container with ID starting with ba476cc77685193cf166d7a7d8c199d1d31a1e9284d6a4867ac62f8aa7c59599 not found: ID does not exist"
Sep 18 13:29:38.402456 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.402386    2566 scope.go:117] "RemoveContainer" containerID="45db1f9749b53cfadc53417ded5de1aedeb00402013f7a0c77de7d9f327ffe38"
Sep 18 13:29:38.403137 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:29:38.403109    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45db1f9749b53cfadc53417ded5de1aedeb00402013f7a0c77de7d9f327ffe38\": container with ID starting with 45db1f9749b53cfadc53417ded5de1aedeb00402013f7a0c77de7d9f327ffe38 not found: ID does not exist" containerID="45db1f9749b53cfadc53417ded5de1aedeb00402013f7a0c77de7d9f327ffe38"
Sep 18 13:29:38.403213 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.403147    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45db1f9749b53cfadc53417ded5de1aedeb00402013f7a0c77de7d9f327ffe38"} err="failed to get container status \"45db1f9749b53cfadc53417ded5de1aedeb00402013f7a0c77de7d9f327ffe38\": rpc error: code = NotFound desc = could not find container \"45db1f9749b53cfadc53417ded5de1aedeb00402013f7a0c77de7d9f327ffe38\": container with ID starting with 45db1f9749b53cfadc53417ded5de1aedeb00402013f7a0c77de7d9f327ffe38 not found: ID does not exist"
Sep 18 13:29:38.509063 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.509023    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-workload-socket\") pod \"32c7f112-cb93-405d-baab-8b0d9c28c54f\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") "
Sep 18 13:29:38.509263 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.509090    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32c7f112-cb93-405d-baab-8b0d9c28c54f-istio-token\") pod \"32c7f112-cb93-405d-baab-8b0d9c28c54f\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") "
Sep 18 13:29:38.509263 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.509136    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tk7j\" (UniqueName: \"kubernetes.io/projected/32c7f112-cb93-405d-baab-8b0d9c28c54f-kube-api-access-5tk7j\") pod \"32c7f112-cb93-405d-baab-8b0d9c28c54f\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") "
Sep 18 13:29:38.509263 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.509163    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-workload-certs\") pod \"32c7f112-cb93-405d-baab-8b0d9c28c54f\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") "
Sep 18 13:29:38.509263 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.509201    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32c7f112-cb93-405d-baab-8b0d9c28c54f-istio-podinfo\") pod \"32c7f112-cb93-405d-baab-8b0d9c28c54f\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") "
Sep 18 13:29:38.509487 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.509341    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "32c7f112-cb93-405d-baab-8b0d9c28c54f" (UID: "32c7f112-cb93-405d-baab-8b0d9c28c54f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:29:38.509487 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.509353    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-istio-envoy\") pod \"32c7f112-cb93-405d-baab-8b0d9c28c54f\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") "
Sep 18 13:29:38.509487 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.509435    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "32c7f112-cb93-405d-baab-8b0d9c28c54f" (UID: "32c7f112-cb93-405d-baab-8b0d9c28c54f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:29:38.509487 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.509469    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-istio-data\") pod \"32c7f112-cb93-405d-baab-8b0d9c28c54f\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") "
Sep 18 13:29:38.509725 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.509530    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32c7f112-cb93-405d-baab-8b0d9c28c54f-istiod-ca-cert\") pod \"32c7f112-cb93-405d-baab-8b0d9c28c54f\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") "
Sep 18 13:29:38.509725 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.509579    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-credential-socket\") pod \"32c7f112-cb93-405d-baab-8b0d9c28c54f\" (UID: \"32c7f112-cb93-405d-baab-8b0d9c28c54f\") "
Sep 18 13:29:38.509820 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.509578    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-istio-data" (OuterVolumeSpecName: "istio-data") pod "32c7f112-cb93-405d-baab-8b0d9c28c54f" (UID: "32c7f112-cb93-405d-baab-8b0d9c28c54f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:29:38.509899 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.509857    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c7f112-cb93-405d-baab-8b0d9c28c54f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "32c7f112-cb93-405d-baab-8b0d9c28c54f" (UID: "32c7f112-cb93-405d-baab-8b0d9c28c54f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:29:38.509899 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.509800    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "32c7f112-cb93-405d-baab-8b0d9c28c54f" (UID: "32c7f112-cb93-405d-baab-8b0d9c28c54f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:29:38.510136 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.510116    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32c7f112-cb93-405d-baab-8b0d9c28c54f-istiod-ca-cert\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:29:38.510198 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.510142    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-credential-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:29:38.510198 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.510157    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-workload-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:29:38.510198 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.510171    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-workload-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:29:38.510198 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.510186    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-istio-data\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:29:38.511747 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.511726    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "32c7f112-cb93-405d-baab-8b0d9c28c54f" (UID: "32c7f112-cb93-405d-baab-8b0d9c28c54f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:29:38.511840 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.511822    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c7f112-cb93-405d-baab-8b0d9c28c54f-istio-token" (OuterVolumeSpecName: "istio-token") pod "32c7f112-cb93-405d-baab-8b0d9c28c54f" (UID: "32c7f112-cb93-405d-baab-8b0d9c28c54f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:29:38.511930 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.511910    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32c7f112-cb93-405d-baab-8b0d9c28c54f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "32c7f112-cb93-405d-baab-8b0d9c28c54f" (UID: "32c7f112-cb93-405d-baab-8b0d9c28c54f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 13:29:38.512204 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.512185    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c7f112-cb93-405d-baab-8b0d9c28c54f-kube-api-access-5tk7j" (OuterVolumeSpecName: "kube-api-access-5tk7j") pod "32c7f112-cb93-405d-baab-8b0d9c28c54f" (UID: "32c7f112-cb93-405d-baab-8b0d9c28c54f"). InnerVolumeSpecName "kube-api-access-5tk7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:29:38.611034 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.610993    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32c7f112-cb93-405d-baab-8b0d9c28c54f-istio-token\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:29:38.611034 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.611026    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tk7j\" (UniqueName: \"kubernetes.io/projected/32c7f112-cb93-405d-baab-8b0d9c28c54f-kube-api-access-5tk7j\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:29:38.611034 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.611037    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32c7f112-cb93-405d-baab-8b0d9c28c54f-istio-podinfo\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:29:38.611034 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.611046    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32c7f112-cb93-405d-baab-8b0d9c28c54f-istio-envoy\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:29:38.703893 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.703865    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"]
Sep 18 13:29:38.708459 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:38.708432    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a304aa20-openshift-default-587d4d7b44-7nzxv"]
Sep 18 13:29:39.101670 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:39.101638    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:39.102458 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:39.102433    2566 patch_prober.go:28] interesting pod/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 18 13:29:39.102570 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:39.102482    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb" podUID="03837341-329a-4b27-827e-4b95a242d96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 18 13:29:40.077648 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:40.077618    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32c7f112-cb93-405d-baab-8b0d9c28c54f" path="/var/lib/kubelet/pods/32c7f112-cb93-405d-baab-8b0d9c28c54f/volumes"
Sep 18 13:29:40.102403 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:40.102374    2566 patch_prober.go:28] interesting pod/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 18 13:29:40.102550 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:40.102452    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb" podUID="03837341-329a-4b27-827e-4b95a242d96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 18 13:29:40.135326 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:40.135290    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:40.157869 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:40.157823    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62"]
Sep 18 13:29:40.158644 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:40.158610    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32c7f112-cb93-405d-baab-8b0d9c28c54f" containerName="istio-proxy"
Sep 18 13:29:40.158781 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:40.158647    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="32c7f112-cb93-405d-baab-8b0d9c28c54f" containerName="istio-proxy"
Sep 18 13:29:40.158841 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:40.158800    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="32c7f112-cb93-405d-baab-8b0d9c28c54f" containerName="istio-proxy"
Sep 18 13:29:40.191478 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:40.191447    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62"
Sep 18 13:29:40.194974 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:40.194521    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-8178145c\""
Sep 18 13:29:40.196831 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:40.196804    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62"]
Sep 18 13:29:40.331132 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:40.331054    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3430d9af-56e3-430a-853c-6c8283e9aee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62\" (UID: \"3430d9af-56e3-430a-853c-6c8283e9aee2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62"
Sep 18 13:29:40.331132 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:40.331092    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmfwj\" (UniqueName: \"kubernetes.io/projected/3430d9af-56e3-430a-853c-6c8283e9aee2-kube-api-access-gmfwj\") pod \"payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62\" (UID: \"3430d9af-56e3-430a-853c-6c8283e9aee2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62"
Sep 18 13:29:40.431868 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:40.431825    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3430d9af-56e3-430a-853c-6c8283e9aee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62\" (UID: \"3430d9af-56e3-430a-853c-6c8283e9aee2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62"
Sep 18 13:29:40.431868 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:40.431870    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmfwj\" (UniqueName: \"kubernetes.io/projected/3430d9af-56e3-430a-853c-6c8283e9aee2-kube-api-access-gmfwj\") pod \"payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62\" (UID: \"3430d9af-56e3-430a-853c-6c8283e9aee2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62"
Sep 18 13:29:40.432533 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:40.432513    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3430d9af-56e3-430a-853c-6c8283e9aee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62\" (UID: \"3430d9af-56e3-430a-853c-6c8283e9aee2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62"
Sep 18 13:29:40.451877 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:40.451824    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmfwj\" (UniqueName: \"kubernetes.io/projected/3430d9af-56e3-430a-853c-6c8283e9aee2-kube-api-access-gmfwj\") pod \"payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62\" (UID: \"3430d9af-56e3-430a-853c-6c8283e9aee2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62"
Sep 18 13:29:40.505605 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:40.505555    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62"
Sep 18 13:29:40.819700 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:40.819671    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62"]
Sep 18 13:29:40.821518 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:29:40.821474    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3430d9af_56e3_430a_853c_6c8283e9aee2.slice/crio-cdb40dcd8ee56643b1f7c8cf02b9275c53a93180bbf2270d2c60ceb6ebe87438 WatchSource:0}: Error finding container cdb40dcd8ee56643b1f7c8cf02b9275c53a93180bbf2270d2c60ceb6ebe87438: Status 404 returned error can't find the container with id cdb40dcd8ee56643b1f7c8cf02b9275c53a93180bbf2270d2c60ceb6ebe87438
Sep 18 13:29:41.102817 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:41.102781    2566 patch_prober.go:28] interesting pod/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 18 13:29:41.103202 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:41.102837    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb" podUID="03837341-329a-4b27-827e-4b95a242d96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 18 13:29:41.399178 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:41.399083    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62" event={"ID":"3430d9af-56e3-430a-853c-6c8283e9aee2","Type":"ContainerStarted","Data":"baf896f9a5ec3b9d44e3f40e9df070bc0ec6564f1ccfd0095a0d439fc271cd74"}
Sep 18 13:29:41.399178 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:41.399134    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62" event={"ID":"3430d9af-56e3-430a-853c-6c8283e9aee2","Type":"ContainerStarted","Data":"cdb40dcd8ee56643b1f7c8cf02b9275c53a93180bbf2270d2c60ceb6ebe87438"}
Sep 18 13:29:41.399364 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:41.399224    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62"
Sep 18 13:29:41.419067 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:41.419012    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62" podStartSLOduration=1.418996069 podStartE2EDuration="1.418996069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:29:41.416441773 +0000 UTC m=+889.851318176" watchObservedRunningTime="2026-09-18 13:29:41.418996069 +0000 UTC m=+889.853872463"
Sep 18 13:29:42.102773 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:42.102730    2566 patch_prober.go:28] interesting pod/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 18 13:29:42.102953 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:42.102791    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb" podUID="03837341-329a-4b27-827e-4b95a242d96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 18 13:29:43.102843 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:43.102805    2566 patch_prober.go:28] interesting pod/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 18 13:29:43.103002 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:43.102859    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb" podUID="03837341-329a-4b27-827e-4b95a242d96e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 18 13:29:44.107284 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:44.107256    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:44.108206 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:44.108187    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:29:52.013093 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:52.013065    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k86vn_55e33e14-3f7a-4793-a724-9fd5022740d7/console-operator/1.log"
Sep 18 13:29:52.016106 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:52.016078    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k86vn_55e33e14-3f7a-4793-a724-9fd5022740d7/console-operator/1.log"
Sep 18 13:29:52.406111 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:29:52.406082    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62"
Sep 18 13:30:00.172045 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:00.172009    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8178145c-29828970-44vv8"]
Sep 18 13:30:00.172453 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:00.172438    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32c7f112-cb93-405d-baab-8b0d9c28c54f" containerName="istio-proxy"
Sep 18 13:30:00.172491 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:00.172454    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="32c7f112-cb93-405d-baab-8b0d9c28c54f" containerName="istio-proxy"
Sep 18 13:30:00.172551 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:00.172520    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="32c7f112-cb93-405d-baab-8b0d9c28c54f" containerName="istio-proxy"
Sep 18 13:30:00.176987 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:00.176964    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828970-xnrhx"]
Sep 18 13:30:00.177111 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:00.177084    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8178145c-29828970-44vv8"
Sep 18 13:30:00.180553 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:00.180530    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nh2cd\""
Sep 18 13:30:00.180702 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:00.180684    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828970-nfv72"]
Sep 18 13:30:00.180795 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:00.180778    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828970-xnrhx"
Sep 18 13:30:00.184292 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:00.184273    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828970-nfv72"
Sep 18 13:30:00.188348 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:00.188332    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8178145c-29828970-44vv8"
Sep 18 13:30:00.192723 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:00.192679    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8178145c-29828970-44vv8"]
Sep 18 13:30:00.198663 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:00.198636    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828970-xnrhx"
Sep 18 13:30:00.205508 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:00.205481    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828970-xnrhx"]
Sep 18 13:30:00.227289 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:00.227256    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828970-nfv72"]
Sep 18 13:30:00.231165 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:00.231128    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34aeca-29828970-zntzb"]
Sep 18 13:30:00.237709 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:00.237679    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34aeca-29828970-zntzb"
Sep 18 13:30:00.240191 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:00.240166    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34aeca-29828970-zntzb"]
Sep 18 13:30:00.253184 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:00.253157    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828970-nfv72"
Sep 18 13:30:00.298359 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:00.298144    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34aeca-29828970-zntzb"
Sep 18 13:30:00.489441 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:00.489413    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8178145c-29828970-44vv8"]
Sep 18 13:30:00.491451 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:30:00.491412    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6757c54a_c913_4c89_8d2b_50db7d3a2443.slice/crio-9b28e70a02dcd584c334f064b23b3cd0fa87c1e60eef127de20a5f7c1cc40d6f WatchSource:0}: Error finding container 9b28e70a02dcd584c334f064b23b3cd0fa87c1e60eef127de20a5f7c1cc40d6f: Status 404 returned error can't find the container with id 9b28e70a02dcd584c334f064b23b3cd0fa87c1e60eef127de20a5f7c1cc40d6f
Sep 18 13:30:00.515095 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:00.515070    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828970-xnrhx"]
Sep 18 13:30:00.528812 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:30:00.528787    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1370e08f_2628_4be3_90ac_4568d17d1bb3.slice/crio-6bd115004987649f1dce6f348fefb7b564efe4732c848854b127d2ed4b7c22d0 WatchSource:0}: Error finding container 6bd115004987649f1dce6f348fefb7b564efe4732c848854b127d2ed4b7c22d0: Status 404 returned error can't find the container with id 6bd115004987649f1dce6f348fefb7b564efe4732c848854b127d2ed4b7c22d0
Sep 18 13:30:00.556573 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:00.556549    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828970-nfv72"]
Sep 18 13:30:00.560131 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:30:00.559993    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4511b800_23d4_457b_a5bc_99c3d2605431.slice/crio-e205e2991b39aff6fabd8fac3fe9a031247799ab9623d8a9590713a50d454004 WatchSource:0}: Error finding container e205e2991b39aff6fabd8fac3fe9a031247799ab9623d8a9590713a50d454004: Status 404 returned error can't find the container with id e205e2991b39aff6fabd8fac3fe9a031247799ab9623d8a9590713a50d454004
Sep 18 13:30:00.593609 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:00.593572    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34aeca-29828970-zntzb"]
Sep 18 13:30:00.597905 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:30:00.597880    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5529278f_7402_47a8_89bb_2aa69d59f6bb.slice/crio-dd49c93571a52f595d3a2e0968f60587540cfe77bba5b8041f02c44c98fdc2bf WatchSource:0}: Error finding container dd49c93571a52f595d3a2e0968f60587540cfe77bba5b8041f02c44c98fdc2bf: Status 404 returned error can't find the container with id dd49c93571a52f595d3a2e0968f60587540cfe77bba5b8041f02c44c98fdc2bf
Sep 18 13:30:01.502404 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:01.502362    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34aeca-29828970-zntzb" event={"ID":"5529278f-7402-47a8-89bb-2aa69d59f6bb","Type":"ContainerStarted","Data":"19ceabdc1eae44ad206b57f859affea7c8dc374b89a4c8da5befb8d414bb7b00"}
Sep 18 13:30:01.502889 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:01.502428    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34aeca-29828970-zntzb" event={"ID":"5529278f-7402-47a8-89bb-2aa69d59f6bb","Type":"ContainerStarted","Data":"dd49c93571a52f595d3a2e0968f60587540cfe77bba5b8041f02c44c98fdc2bf"}
Sep 18 13:30:01.504019 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:01.503987    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828970-nfv72" event={"ID":"4511b800-23d4-457b-a5bc-99c3d2605431","Type":"ContainerStarted","Data":"a8796d63ed24bd9055381ec04d05ce65069ca4526e895ac55877e6cc54925ce2"}
Sep 18 13:30:01.504136 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:01.504042    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828970-nfv72" event={"ID":"4511b800-23d4-457b-a5bc-99c3d2605431","Type":"ContainerStarted","Data":"e205e2991b39aff6fabd8fac3fe9a031247799ab9623d8a9590713a50d454004"}
Sep 18 13:30:01.505743 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:01.505660    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828970-xnrhx" event={"ID":"1370e08f-2628-4be3-90ac-4568d17d1bb3","Type":"ContainerStarted","Data":"56dd882afbf02ef63999761094326cf4f83f451cbc0189729ae7f568f2cd0302"}
Sep 18 13:30:01.505743 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:01.505714    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828970-xnrhx" event={"ID":"1370e08f-2628-4be3-90ac-4568d17d1bb3","Type":"ContainerStarted","Data":"6bd115004987649f1dce6f348fefb7b564efe4732c848854b127d2ed4b7c22d0"}
Sep 18 13:30:01.507308 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:01.507284    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8178145c-29828970-44vv8" event={"ID":"6757c54a-c913-4c89-8d2b-50db7d3a2443","Type":"ContainerStarted","Data":"0f71a65fae577f2b4469d84e41bcf887bfa1dfb2be0f3d07ff80c913391d6c73"}
Sep 18 13:30:01.507409 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:01.507330    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8178145c-29828970-44vv8" event={"ID":"6757c54a-c913-4c89-8d2b-50db7d3a2443","Type":"ContainerStarted","Data":"9b28e70a02dcd584c334f064b23b3cd0fa87c1e60eef127de20a5f7c1cc40d6f"}
Sep 18 13:30:01.520477 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:01.520426    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34aeca-29828970-zntzb" podStartSLOduration=1.520411921 podStartE2EDuration="1.520411921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:30:01.519825477 +0000 UTC m=+909.954701881" watchObservedRunningTime="2026-09-18 13:30:01.520411921 +0000 UTC m=+909.955288326"
Sep 18 13:30:01.590883 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:01.590827    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828970-xnrhx" podStartSLOduration=1.590810847 podStartE2EDuration="1.590810847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:30:01.554243235 +0000 UTC m=+909.989119642" watchObservedRunningTime="2026-09-18 13:30:01.590810847 +0000 UTC m=+910.025687250"
Sep 18 13:30:01.591525 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:01.591495    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828970-nfv72" podStartSLOduration=1.591486816 podStartE2EDuration="1.591486816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:30:01.590091062 +0000 UTC m=+910.024967465" watchObservedRunningTime="2026-09-18 13:30:01.591486816 +0000 UTC m=+910.026363214"
Sep 18 13:30:01.626875 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:01.626807    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8178145c-29828970-44vv8" podStartSLOduration=1.626784698 podStartE2EDuration="1.626784698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:30:01.621866692 +0000 UTC m=+910.056743097" watchObservedRunningTime="2026-09-18 13:30:01.626784698 +0000 UTC m=+910.061661107"
Sep 18 13:30:02.515136 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:02.515099    2566 generic.go:356] "Generic (PLEG): container finished" podID="4511b800-23d4-457b-a5bc-99c3d2605431" containerID="a8796d63ed24bd9055381ec04d05ce65069ca4526e895ac55877e6cc54925ce2" exitCode=0
Sep 18 13:30:02.515550 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:02.515181    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828970-nfv72" event={"ID":"4511b800-23d4-457b-a5bc-99c3d2605431","Type":"ContainerDied","Data":"a8796d63ed24bd9055381ec04d05ce65069ca4526e895ac55877e6cc54925ce2"}
Sep 18 13:30:02.516743 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:02.516721    2566 generic.go:356] "Generic (PLEG): container finished" podID="1370e08f-2628-4be3-90ac-4568d17d1bb3" containerID="56dd882afbf02ef63999761094326cf4f83f451cbc0189729ae7f568f2cd0302" exitCode=0
Sep 18 13:30:02.516863 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:02.516786    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828970-xnrhx" event={"ID":"1370e08f-2628-4be3-90ac-4568d17d1bb3","Type":"ContainerDied","Data":"56dd882afbf02ef63999761094326cf4f83f451cbc0189729ae7f568f2cd0302"}
Sep 18 13:30:02.518276 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:02.518251    2566 generic.go:356] "Generic (PLEG): container finished" podID="6757c54a-c913-4c89-8d2b-50db7d3a2443" containerID="0f71a65fae577f2b4469d84e41bcf887bfa1dfb2be0f3d07ff80c913391d6c73" exitCode=0
Sep 18 13:30:02.518385 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:02.518361    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8178145c-29828970-44vv8" event={"ID":"6757c54a-c913-4c89-8d2b-50db7d3a2443","Type":"ContainerDied","Data":"0f71a65fae577f2b4469d84e41bcf887bfa1dfb2be0f3d07ff80c913391d6c73"}
Sep 18 13:30:02.519826 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:02.519806    2566 generic.go:356] "Generic (PLEG): container finished" podID="5529278f-7402-47a8-89bb-2aa69d59f6bb" containerID="19ceabdc1eae44ad206b57f859affea7c8dc374b89a4c8da5befb8d414bb7b00" exitCode=0
Sep 18 13:30:02.519922 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:02.519849    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34aeca-29828970-zntzb" event={"ID":"5529278f-7402-47a8-89bb-2aa69d59f6bb","Type":"ContainerDied","Data":"19ceabdc1eae44ad206b57f859affea7c8dc374b89a4c8da5befb8d414bb7b00"}
Sep 18 13:30:03.707831 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:03.707583    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828970-xnrhx"
Sep 18 13:30:03.795411 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:03.795382    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34aeca-29828970-zntzb"
Sep 18 13:30:03.799349 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:03.799252    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828970-nfv72"
Sep 18 13:30:03.803516 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:03.803378    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8178145c-29828970-44vv8"
Sep 18 13:30:04.532511 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:04.532481    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34aeca-29828970-zntzb"
Sep 18 13:30:04.532511 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:04.532504    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34aeca-29828970-zntzb" event={"ID":"5529278f-7402-47a8-89bb-2aa69d59f6bb","Type":"ContainerDied","Data":"dd49c93571a52f595d3a2e0968f60587540cfe77bba5b8041f02c44c98fdc2bf"}
Sep 18 13:30:04.532760 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:04.532550    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd49c93571a52f595d3a2e0968f60587540cfe77bba5b8041f02c44c98fdc2bf"
Sep 18 13:30:04.533897 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:04.533880    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828970-nfv72"
Sep 18 13:30:04.534030 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:04.533907    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828970-nfv72" event={"ID":"4511b800-23d4-457b-a5bc-99c3d2605431","Type":"ContainerDied","Data":"e205e2991b39aff6fabd8fac3fe9a031247799ab9623d8a9590713a50d454004"}
Sep 18 13:30:04.534030 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:04.533959    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e205e2991b39aff6fabd8fac3fe9a031247799ab9623d8a9590713a50d454004"
Sep 18 13:30:04.535348 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:04.535328    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828970-xnrhx"
Sep 18 13:30:04.535441 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:04.535349    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828970-xnrhx" event={"ID":"1370e08f-2628-4be3-90ac-4568d17d1bb3","Type":"ContainerDied","Data":"6bd115004987649f1dce6f348fefb7b564efe4732c848854b127d2ed4b7c22d0"}
Sep 18 13:30:04.535441 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:04.535387    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bd115004987649f1dce6f348fefb7b564efe4732c848854b127d2ed4b7c22d0"
Sep 18 13:30:04.536770 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:04.536756    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8178145c-29828970-44vv8"
Sep 18 13:30:04.536770 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:04.536761    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8178145c-29828970-44vv8" event={"ID":"6757c54a-c913-4c89-8d2b-50db7d3a2443","Type":"ContainerDied","Data":"9b28e70a02dcd584c334f064b23b3cd0fa87c1e60eef127de20a5f7c1cc40d6f"}
Sep 18 13:30:04.536910 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:04.536801    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b28e70a02dcd584c334f064b23b3cd0fa87c1e60eef127de20a5f7c1cc40d6f"
Sep 18 13:30:06.243552 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:06.243508    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8178145c-kvtsn"]
Sep 18 13:30:06.244168 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:06.244144    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6757c54a-c913-4c89-8d2b-50db7d3a2443" containerName="cleanup"
Sep 18 13:30:06.244255 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:06.244170    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6757c54a-c913-4c89-8d2b-50db7d3a2443" containerName="cleanup"
Sep 18 13:30:06.244255 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:06.244185    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1370e08f-2628-4be3-90ac-4568d17d1bb3" containerName="cleanup"
Sep 18 13:30:06.244255 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:06.244194    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1370e08f-2628-4be3-90ac-4568d17d1bb3" containerName="cleanup"
Sep 18 13:30:06.244255 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:06.244226    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4511b800-23d4-457b-a5bc-99c3d2605431" containerName="cleanup"
Sep 18 13:30:06.244255 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:06.244234    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4511b800-23d4-457b-a5bc-99c3d2605431" containerName="cleanup"
Sep 18 13:30:06.244255 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:06.244252    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5529278f-7402-47a8-89bb-2aa69d59f6bb" containerName="cleanup"
Sep 18 13:30:06.244582 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:06.244262    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5529278f-7402-47a8-89bb-2aa69d59f6bb" containerName="cleanup"
Sep 18 13:30:06.244582 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:06.244361    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5529278f-7402-47a8-89bb-2aa69d59f6bb" containerName="cleanup"
Sep 18 13:30:06.244582 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:06.244377    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6757c54a-c913-4c89-8d2b-50db7d3a2443" containerName="cleanup"
Sep 18 13:30:06.244582 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:06.244387    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1370e08f-2628-4be3-90ac-4568d17d1bb3" containerName="cleanup"
Sep 18 13:30:06.244582 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:06.244397    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4511b800-23d4-457b-a5bc-99c3d2605431" containerName="cleanup"
Sep 18 13:30:06.248199 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:06.248172    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8178145c-kvtsn"
Sep 18 13:30:06.251804 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:06.251784    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nh2cd\""
Sep 18 13:30:06.252106 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:06.252084    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:30:06.279805 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:06.279778    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8178145c-kvtsn"]
Sep 18 13:30:06.315773 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:06.315738    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/895a5ffb-131b-495c-871b-0de6163049f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8178145c-kvtsn\" (UID: \"895a5ffb-131b-495c-871b-0de6163049f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8178145c-kvtsn"
Sep 18 13:30:06.416656 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:06.416620    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/895a5ffb-131b-495c-871b-0de6163049f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8178145c-kvtsn\" (UID: \"895a5ffb-131b-495c-871b-0de6163049f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8178145c-kvtsn"
Sep 18 13:30:06.417279 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:06.417257    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/895a5ffb-131b-495c-871b-0de6163049f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8178145c-kvtsn\" (UID: \"895a5ffb-131b-495c-871b-0de6163049f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8178145c-kvtsn"
Sep 18 13:30:06.564284 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:06.564183    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8178145c-kvtsn"
Sep 18 13:30:06.845859 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:06.845790    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8178145c-kvtsn"]
Sep 18 13:30:06.848175 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:30:06.848144    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895a5ffb_131b_495c_871b_0de6163049f7.slice/crio-946bf00d75c51f70f2a5f4e090d6d5f5d6a8ecdd4a15ba540e07ae920e9da0b8 WatchSource:0}: Error finding container 946bf00d75c51f70f2a5f4e090d6d5f5d6a8ecdd4a15ba540e07ae920e9da0b8: Status 404 returned error can't find the container with id 946bf00d75c51f70f2a5f4e090d6d5f5d6a8ecdd4a15ba540e07ae920e9da0b8
Sep 18 13:30:07.561780 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:07.561747    2566 generic.go:356] "Generic (PLEG): container finished" podID="895a5ffb-131b-495c-871b-0de6163049f7" containerID="23c37de3b316add730d3ff03e7b3311d58606f4dd28377bee71d9ed1e487ca00" exitCode=0
Sep 18 13:30:07.562134 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:07.561792    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8178145c-kvtsn" event={"ID":"895a5ffb-131b-495c-871b-0de6163049f7","Type":"ContainerDied","Data":"23c37de3b316add730d3ff03e7b3311d58606f4dd28377bee71d9ed1e487ca00"}
Sep 18 13:30:07.562134 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:07.561833    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8178145c-kvtsn" event={"ID":"895a5ffb-131b-495c-871b-0de6163049f7","Type":"ContainerStarted","Data":"946bf00d75c51f70f2a5f4e090d6d5f5d6a8ecdd4a15ba540e07ae920e9da0b8"}
Sep 18 13:30:08.718254 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:08.718223    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8178145c-kvtsn"
Sep 18 13:30:08.837687 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:08.837609    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/895a5ffb-131b-495c-871b-0de6163049f7-maas-api-service-ca\") pod \"895a5ffb-131b-495c-871b-0de6163049f7\" (UID: \"895a5ffb-131b-495c-871b-0de6163049f7\") "
Sep 18 13:30:08.838016 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:08.837994    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/895a5ffb-131b-495c-871b-0de6163049f7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "895a5ffb-131b-495c-871b-0de6163049f7" (UID: "895a5ffb-131b-495c-871b-0de6163049f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:30:08.938793 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:08.938754    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/895a5ffb-131b-495c-871b-0de6163049f7-maas-api-service-ca\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:30:09.580609 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:09.580571    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8178145c-kvtsn"
Sep 18 13:30:09.580792 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:09.580566    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8178145c-kvtsn" event={"ID":"895a5ffb-131b-495c-871b-0de6163049f7","Type":"ContainerDied","Data":"946bf00d75c51f70f2a5f4e090d6d5f5d6a8ecdd4a15ba540e07ae920e9da0b8"}
Sep 18 13:30:09.580792 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:09.580718    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="946bf00d75c51f70f2a5f4e090d6d5f5d6a8ecdd4a15ba540e07ae920e9da0b8"
Sep 18 13:30:10.587640 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:10.587586    2566 generic.go:356] "Generic (PLEG): container finished" podID="6f42433c-819f-40ef-a520-df3a5e888160" containerID="88cd44a4b0b08698a9a020c4a90e7dd57a020538c4faea5c404d88f6d258b048" exitCode=0
Sep 18 13:30:10.588082 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:10.587663    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz" event={"ID":"6f42433c-819f-40ef-a520-df3a5e888160","Type":"ContainerDied","Data":"88cd44a4b0b08698a9a020c4a90e7dd57a020538c4faea5c404d88f6d258b048"}
Sep 18 13:30:10.588141 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:10.588114    2566 scope.go:117] "RemoveContainer" containerID="88cd44a4b0b08698a9a020c4a90e7dd57a020538c4faea5c404d88f6d258b048"
Sep 18 13:30:10.589278 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:10.589251    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:30:10.589364 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:10.589319    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:30:10.589364 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:10.589350    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:30:11.595780 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:11.595744    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz" event={"ID":"6f42433c-819f-40ef-a520-df3a5e888160","Type":"ContainerStarted","Data":"9148f21a4cd1b2b80a25f7fff2e57643f1def8e0535b786cc5244fcda86433c8"}
Sep 18 13:30:12.494671 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:12.494635    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:30:12.494844 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:12.494808    2566 patch_prober.go:28] interesting pod/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 18 13:30:12.494889 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:12.494847    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz" podUID="6f42433c-819f-40ef-a520-df3a5e888160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 18 13:30:13.494561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:13.494522    2566 patch_prober.go:28] interesting pod/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 18 13:30:13.494979 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:13.494578    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz" podUID="6f42433c-819f-40ef-a520-df3a5e888160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 18 13:30:14.494144 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:14.494098    2566 patch_prober.go:28] interesting pod/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 18 13:30:14.494328 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:14.494172    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz" podUID="6f42433c-819f-40ef-a520-df3a5e888160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 18 13:30:15.494748 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:15.494707    2566 patch_prober.go:28] interesting pod/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 18 13:30:15.495150 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:15.494765    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz" podUID="6f42433c-819f-40ef-a520-df3a5e888160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 18 13:30:16.494314 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:16.494278    2566 patch_prober.go:28] interesting pod/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 18 13:30:16.494503 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:16.494333    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz" podUID="6f42433c-819f-40ef-a520-df3a5e888160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 18 13:30:17.498280 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:17.498249    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:30:17.498707 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:17.498530    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:30:17.499399 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:17.499377    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:30:17.536789 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:17.536750    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8178145c-29828970-44vv8"]
Sep 18 13:30:17.540735 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:17.540701    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-8178145c-29828970-44vv8"]
Sep 18 13:30:17.607989 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:17.607952    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62"]
Sep 18 13:30:17.608184 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:17.608162    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62" podUID="3430d9af-56e3-430a-853c-6c8283e9aee2" containerName="payload-pre-processing" containerID="cri-o://baf896f9a5ec3b9d44e3f40e9df070bc0ec6564f1ccfd0095a0d439fc271cd74" gracePeriod=30
Sep 18 13:30:17.864241 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:17.864217    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62"
Sep 18 13:30:17.901352 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:17.901321    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8178145c-kvtsn"]
Sep 18 13:30:17.906763 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:17.906733    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8178145c-kvtsn"]
Sep 18 13:30:17.929059 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:17.929017    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3430d9af-56e3-430a-853c-6c8283e9aee2-plugins-config-volume\") pod \"3430d9af-56e3-430a-853c-6c8283e9aee2\" (UID: \"3430d9af-56e3-430a-853c-6c8283e9aee2\") "
Sep 18 13:30:17.929272 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:17.929126    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmfwj\" (UniqueName: \"kubernetes.io/projected/3430d9af-56e3-430a-853c-6c8283e9aee2-kube-api-access-gmfwj\") pod \"3430d9af-56e3-430a-853c-6c8283e9aee2\" (UID: \"3430d9af-56e3-430a-853c-6c8283e9aee2\") "
Sep 18 13:30:17.930758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:17.930682    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3430d9af-56e3-430a-853c-6c8283e9aee2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3430d9af-56e3-430a-853c-6c8283e9aee2" (UID: "3430d9af-56e3-430a-853c-6c8283e9aee2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:30:17.936615 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:17.936554    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3430d9af-56e3-430a-853c-6c8283e9aee2-kube-api-access-gmfwj" (OuterVolumeSpecName: "kube-api-access-gmfwj") pod "3430d9af-56e3-430a-853c-6c8283e9aee2" (UID: "3430d9af-56e3-430a-853c-6c8283e9aee2"). InnerVolumeSpecName "kube-api-access-gmfwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:30:18.030935 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:18.030896    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3430d9af-56e3-430a-853c-6c8283e9aee2-plugins-config-volume\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:30:18.030935 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:18.030940    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmfwj\" (UniqueName: \"kubernetes.io/projected/3430d9af-56e3-430a-853c-6c8283e9aee2-kube-api-access-gmfwj\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:30:18.052896 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:18.052809    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"]
Sep 18 13:30:18.079440 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:18.079395    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6757c54a-c913-4c89-8d2b-50db7d3a2443" path="/var/lib/kubelet/pods/6757c54a-c913-4c89-8d2b-50db7d3a2443/volumes"
Sep 18 13:30:18.079977 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:18.079947    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="895a5ffb-131b-495c-871b-0de6163049f7" path="/var/lib/kubelet/pods/895a5ffb-131b-495c-871b-0de6163049f7/volumes"
Sep 18 13:30:18.643375 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:18.643344    2566 generic.go:356] "Generic (PLEG): container finished" podID="3430d9af-56e3-430a-853c-6c8283e9aee2" containerID="baf896f9a5ec3b9d44e3f40e9df070bc0ec6564f1ccfd0095a0d439fc271cd74" exitCode=0
Sep 18 13:30:18.643821 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:18.643388    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62"
Sep 18 13:30:18.643821 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:18.643442    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62" event={"ID":"3430d9af-56e3-430a-853c-6c8283e9aee2","Type":"ContainerDied","Data":"baf896f9a5ec3b9d44e3f40e9df070bc0ec6564f1ccfd0095a0d439fc271cd74"}
Sep 18 13:30:18.643821 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:18.643491    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62" event={"ID":"3430d9af-56e3-430a-853c-6c8283e9aee2","Type":"ContainerDied","Data":"cdb40dcd8ee56643b1f7c8cf02b9275c53a93180bbf2270d2c60ceb6ebe87438"}
Sep 18 13:30:18.643821 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:18.643514    2566 scope.go:117] "RemoveContainer" containerID="baf896f9a5ec3b9d44e3f40e9df070bc0ec6564f1ccfd0095a0d439fc271cd74"
Sep 18 13:30:18.643821 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:18.643760    2566 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz" secret="" err="secret \"e2e-migrate-8178145c-gw-openshift-default-dockercfg-m5hd8\" not found"
Sep 18 13:30:18.652509 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:18.652491    2566 scope.go:117] "RemoveContainer" containerID="baf896f9a5ec3b9d44e3f40e9df070bc0ec6564f1ccfd0095a0d439fc271cd74"
Sep 18 13:30:18.652798 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:30:18.652779    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baf896f9a5ec3b9d44e3f40e9df070bc0ec6564f1ccfd0095a0d439fc271cd74\": container with ID starting with baf896f9a5ec3b9d44e3f40e9df070bc0ec6564f1ccfd0095a0d439fc271cd74 not found: ID does not exist" containerID="baf896f9a5ec3b9d44e3f40e9df070bc0ec6564f1ccfd0095a0d439fc271cd74"
Sep 18 13:30:18.652860 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:18.652806    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baf896f9a5ec3b9d44e3f40e9df070bc0ec6564f1ccfd0095a0d439fc271cd74"} err="failed to get container status \"baf896f9a5ec3b9d44e3f40e9df070bc0ec6564f1ccfd0095a0d439fc271cd74\": rpc error: code = NotFound desc = could not find container \"baf896f9a5ec3b9d44e3f40e9df070bc0ec6564f1ccfd0095a0d439fc271cd74\": container with ID starting with baf896f9a5ec3b9d44e3f40e9df070bc0ec6564f1ccfd0095a0d439fc271cd74 not found: ID does not exist"
Sep 18 13:30:18.680612 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:18.679311    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62"]
Sep 18 13:30:18.685878 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:18.685852    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8178145c-58576d8f98-k6g62"]
Sep 18 13:30:18.996187 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:18.996100    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"]
Sep 18 13:30:18.996662 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:18.996643    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3430d9af-56e3-430a-853c-6c8283e9aee2" containerName="payload-pre-processing"
Sep 18 13:30:18.996662 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:18.996662    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3430d9af-56e3-430a-853c-6c8283e9aee2" containerName="payload-pre-processing"
Sep 18 13:30:18.996829 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:18.996674    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="895a5ffb-131b-495c-871b-0de6163049f7" containerName="revoke-keys"
Sep 18 13:30:18.996829 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:18.996681    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="895a5ffb-131b-495c-871b-0de6163049f7" containerName="revoke-keys"
Sep 18 13:30:18.996829 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:18.996781    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="895a5ffb-131b-495c-871b-0de6163049f7" containerName="revoke-keys"
Sep 18 13:30:18.996829 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:18.996793    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3430d9af-56e3-430a-853c-6c8283e9aee2" containerName="payload-pre-processing"
Sep 18 13:30:19.001287 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.001268    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.004061 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.004042    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-92246a6e-openshift-default-dockercfg-xnpbh\""
Sep 18 13:30:19.026716 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.026681    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"]
Sep 18 13:30:19.143207 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.143167    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwdvn\" (UniqueName: \"kubernetes.io/projected/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-kube-api-access-lwdvn\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.143384 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.143227    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istio-token\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.143384 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.143270    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istio-podinfo\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.143384 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.143336    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-credential-socket\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.143384 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.143357    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istio-data\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.143384 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.143377    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istiod-ca-cert\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.143568 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.143400    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-workload-certs\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.143568 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.143473    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-workload-socket\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.143568 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.143517    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istio-envoy\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.244337 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.244299    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwdvn\" (UniqueName: \"kubernetes.io/projected/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-kube-api-access-lwdvn\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.244504 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.244357    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istio-token\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.244504 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.244377    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istio-podinfo\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.244504 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.244419    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-credential-socket\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.244504 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.244436    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istio-data\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.244504 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.244458    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istiod-ca-cert\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.244504 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.244485    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-workload-certs\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.244903 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.244512    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-workload-socket\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.244903 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.244539    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istio-envoy\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.245009 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.244953    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istio-data\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.245009 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.244959    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-workload-certs\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.245114 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.245079    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-workload-socket\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.245114 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.245084    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-credential-socket\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.245409 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.245337    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istiod-ca-cert\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.246946 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.246895    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istio-podinfo\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.246946 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.246918    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istio-envoy\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.261723 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.261695    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istio-token\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.262283 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.262261    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwdvn\" (UniqueName: \"kubernetes.io/projected/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-kube-api-access-lwdvn\") pod \"e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") " pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.311502 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.311469    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:19.599541 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.599507    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"]
Sep 18 13:30:19.601126 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:30:19.601093    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a1789e_8e58_406f_8d4d_bf2528ca5cd5.slice/crio-20cceadab7946f268d403c4cb65a2966721e67b2bd518622796e0fc32437867a WatchSource:0}: Error finding container 20cceadab7946f268d403c4cb65a2966721e67b2bd518622796e0fc32437867a: Status 404 returned error can't find the container with id 20cceadab7946f268d403c4cb65a2966721e67b2bd518622796e0fc32437867a
Sep 18 13:30:19.603444 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.603414    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:30:19.603519 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.603478    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:30:19.603519 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.603507    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:30:19.651081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.651040    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7" event={"ID":"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5","Type":"ContainerStarted","Data":"20cceadab7946f268d403c4cb65a2966721e67b2bd518622796e0fc32437867a"}
Sep 18 13:30:19.652256 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:19.652202    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz" podUID="6f42433c-819f-40ef-a520-df3a5e888160" containerName="istio-proxy" containerID="cri-o://9148f21a4cd1b2b80a25f7fff2e57643f1def8e0535b786cc5244fcda86433c8" gracePeriod=30
Sep 18 13:30:20.078653 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:20.078559    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3430d9af-56e3-430a-853c-6c8283e9aee2" path="/var/lib/kubelet/pods/3430d9af-56e3-430a-853c-6c8283e9aee2/volumes"
Sep 18 13:30:20.659609 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:20.659565    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7" event={"ID":"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5","Type":"ContainerStarted","Data":"f779b1f013aba6bd3cebe8b73c5eb8b0942691780b7642b5bcb48bb23b9ad713"}
Sep 18 13:30:20.686970 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:20.686914    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7" podStartSLOduration=2.68689694 podStartE2EDuration="2.68689694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:30:20.684872405 +0000 UTC m=+929.119748810" watchObservedRunningTime="2026-09-18 13:30:20.68689694 +0000 UTC m=+929.121773344"
Sep 18 13:30:21.312285 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:21.312250    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:21.317630 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:21.317588    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:21.665651 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:21.665619    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:21.666734 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:21.666714    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:24.903342 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:24.903317    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:30:25.005923 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.005835    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f42433c-819f-40ef-a520-df3a5e888160-istiod-ca-cert\") pod \"6f42433c-819f-40ef-a520-df3a5e888160\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") "
Sep 18 13:30:25.005923 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.005906    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-workload-certs\") pod \"6f42433c-819f-40ef-a520-df3a5e888160\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") "
Sep 18 13:30:25.006130 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.005939    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f42433c-819f-40ef-a520-df3a5e888160-istio-podinfo\") pod \"6f42433c-819f-40ef-a520-df3a5e888160\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") "
Sep 18 13:30:25.006130 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.005957    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-istio-envoy\") pod \"6f42433c-819f-40ef-a520-df3a5e888160\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") "
Sep 18 13:30:25.006130 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.006019    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f42433c-819f-40ef-a520-df3a5e888160-istio-token\") pod \"6f42433c-819f-40ef-a520-df3a5e888160\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") "
Sep 18 13:30:25.006130 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.006046    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-istio-data\") pod \"6f42433c-819f-40ef-a520-df3a5e888160\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") "
Sep 18 13:30:25.006130 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.006073    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-workload-socket\") pod \"6f42433c-819f-40ef-a520-df3a5e888160\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") "
Sep 18 13:30:25.006130 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.006109    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpj6b\" (UniqueName: \"kubernetes.io/projected/6f42433c-819f-40ef-a520-df3a5e888160-kube-api-access-dpj6b\") pod \"6f42433c-819f-40ef-a520-df3a5e888160\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") "
Sep 18 13:30:25.006444 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.006168    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-credential-socket\") pod \"6f42433c-819f-40ef-a520-df3a5e888160\" (UID: \"6f42433c-819f-40ef-a520-df3a5e888160\") "
Sep 18 13:30:25.006444 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.006273    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6f42433c-819f-40ef-a520-df3a5e888160" (UID: "6f42433c-819f-40ef-a520-df3a5e888160"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:30:25.006444 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.006345    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f42433c-819f-40ef-a520-df3a5e888160-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6f42433c-819f-40ef-a520-df3a5e888160" (UID: "6f42433c-819f-40ef-a520-df3a5e888160"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:30:25.006640 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.006548    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6f42433c-819f-40ef-a520-df3a5e888160" (UID: "6f42433c-819f-40ef-a520-df3a5e888160"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:30:25.006700 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.006661    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-istio-data" (OuterVolumeSpecName: "istio-data") pod "6f42433c-819f-40ef-a520-df3a5e888160" (UID: "6f42433c-819f-40ef-a520-df3a5e888160"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:30:25.006787 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.006768    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f42433c-819f-40ef-a520-df3a5e888160-istiod-ca-cert\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:30:25.006849 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.006794    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-workload-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:30:25.006849 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.006810    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-istio-data\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:30:25.006849 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.006822    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-credential-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:30:25.006849 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.006770    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6f42433c-819f-40ef-a520-df3a5e888160" (UID: "6f42433c-819f-40ef-a520-df3a5e888160"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:30:25.008364 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.008339    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f42433c-819f-40ef-a520-df3a5e888160-istio-token" (OuterVolumeSpecName: "istio-token") pod "6f42433c-819f-40ef-a520-df3a5e888160" (UID: "6f42433c-819f-40ef-a520-df3a5e888160"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:30:25.008364 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.008353    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f42433c-819f-40ef-a520-df3a5e888160-kube-api-access-dpj6b" (OuterVolumeSpecName: "kube-api-access-dpj6b") pod "6f42433c-819f-40ef-a520-df3a5e888160" (UID: "6f42433c-819f-40ef-a520-df3a5e888160"). InnerVolumeSpecName "kube-api-access-dpj6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:30:25.008584 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.008562    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f42433c-819f-40ef-a520-df3a5e888160-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6f42433c-819f-40ef-a520-df3a5e888160" (UID: "6f42433c-819f-40ef-a520-df3a5e888160"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 13:30:25.008658 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.008567    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6f42433c-819f-40ef-a520-df3a5e888160" (UID: "6f42433c-819f-40ef-a520-df3a5e888160"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:30:25.108293 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.108257    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f42433c-819f-40ef-a520-df3a5e888160-istio-podinfo\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:30:25.108293 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.108286    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-istio-envoy\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:30:25.108293 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.108295    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f42433c-819f-40ef-a520-df3a5e888160-istio-token\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:30:25.108293 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.108302    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f42433c-819f-40ef-a520-df3a5e888160-workload-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:30:25.108551 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.108313    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpj6b\" (UniqueName: \"kubernetes.io/projected/6f42433c-819f-40ef-a520-df3a5e888160-kube-api-access-dpj6b\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:30:25.689671 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.689638    2566 generic.go:356] "Generic (PLEG): container finished" podID="6f42433c-819f-40ef-a520-df3a5e888160" containerID="9148f21a4cd1b2b80a25f7fff2e57643f1def8e0535b786cc5244fcda86433c8" exitCode=0
Sep 18 13:30:25.689847 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.689698    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"
Sep 18 13:30:25.689847 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.689697    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz" event={"ID":"6f42433c-819f-40ef-a520-df3a5e888160","Type":"ContainerDied","Data":"9148f21a4cd1b2b80a25f7fff2e57643f1def8e0535b786cc5244fcda86433c8"}
Sep 18 13:30:25.689847 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.689762    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz" event={"ID":"6f42433c-819f-40ef-a520-df3a5e888160","Type":"ContainerDied","Data":"fa51c43bd6ba1f4ce3064b150ce847973c4d795e2f8959718c34eb5da73d3d80"}
Sep 18 13:30:25.689847 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.689797    2566 scope.go:117] "RemoveContainer" containerID="9148f21a4cd1b2b80a25f7fff2e57643f1def8e0535b786cc5244fcda86433c8"
Sep 18 13:30:25.699671 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.699655    2566 scope.go:117] "RemoveContainer" containerID="88cd44a4b0b08698a9a020c4a90e7dd57a020538c4faea5c404d88f6d258b048"
Sep 18 13:30:25.708388 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.708370    2566 scope.go:117] "RemoveContainer" containerID="9148f21a4cd1b2b80a25f7fff2e57643f1def8e0535b786cc5244fcda86433c8"
Sep 18 13:30:25.708658 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:30:25.708639    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9148f21a4cd1b2b80a25f7fff2e57643f1def8e0535b786cc5244fcda86433c8\": container with ID starting with 9148f21a4cd1b2b80a25f7fff2e57643f1def8e0535b786cc5244fcda86433c8 not found: ID does not exist" containerID="9148f21a4cd1b2b80a25f7fff2e57643f1def8e0535b786cc5244fcda86433c8"
Sep 18 13:30:25.708736 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.708671    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9148f21a4cd1b2b80a25f7fff2e57643f1def8e0535b786cc5244fcda86433c8"} err="failed to get container status \"9148f21a4cd1b2b80a25f7fff2e57643f1def8e0535b786cc5244fcda86433c8\": rpc error: code = NotFound desc = could not find container \"9148f21a4cd1b2b80a25f7fff2e57643f1def8e0535b786cc5244fcda86433c8\": container with ID starting with 9148f21a4cd1b2b80a25f7fff2e57643f1def8e0535b786cc5244fcda86433c8 not found: ID does not exist"
Sep 18 13:30:25.708736 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.708696    2566 scope.go:117] "RemoveContainer" containerID="88cd44a4b0b08698a9a020c4a90e7dd57a020538c4faea5c404d88f6d258b048"
Sep 18 13:30:25.708962 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:30:25.708942    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88cd44a4b0b08698a9a020c4a90e7dd57a020538c4faea5c404d88f6d258b048\": container with ID starting with 88cd44a4b0b08698a9a020c4a90e7dd57a020538c4faea5c404d88f6d258b048 not found: ID does not exist" containerID="88cd44a4b0b08698a9a020c4a90e7dd57a020538c4faea5c404d88f6d258b048"
Sep 18 13:30:25.709007 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.708967    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88cd44a4b0b08698a9a020c4a90e7dd57a020538c4faea5c404d88f6d258b048"} err="failed to get container status \"88cd44a4b0b08698a9a020c4a90e7dd57a020538c4faea5c404d88f6d258b048\": rpc error: code = NotFound desc = could not find container \"88cd44a4b0b08698a9a020c4a90e7dd57a020538c4faea5c404d88f6d258b048\": container with ID starting with 88cd44a4b0b08698a9a020c4a90e7dd57a020538c4faea5c404d88f6d258b048 not found: ID does not exist"
Sep 18 13:30:25.738940 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.738908    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"]
Sep 18 13:30:25.753274 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:25.753243    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-8178145c-gw-openshift-default-bbbd74756-ff2tz"]
Sep 18 13:30:26.078294 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:26.078210    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f42433c-819f-40ef-a520-df3a5e888160" path="/var/lib/kubelet/pods/6f42433c-819f-40ef-a520-df3a5e888160/volumes"
Sep 18 13:30:26.338051 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:26.337964    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6"]
Sep 18 13:30:26.338441 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:26.338422    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f42433c-819f-40ef-a520-df3a5e888160" containerName="istio-proxy"
Sep 18 13:30:26.338539 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:26.338444    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f42433c-819f-40ef-a520-df3a5e888160" containerName="istio-proxy"
Sep 18 13:30:26.338539 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:26.338494    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f42433c-819f-40ef-a520-df3a5e888160" containerName="istio-proxy"
Sep 18 13:30:26.338539 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:26.338504    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f42433c-819f-40ef-a520-df3a5e888160" containerName="istio-proxy"
Sep 18 13:30:26.338726 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:26.338616    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f42433c-819f-40ef-a520-df3a5e888160" containerName="istio-proxy"
Sep 18 13:30:26.343164 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:26.343141    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6"
Sep 18 13:30:26.355523 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:26.355500    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-92246a6e\""
Sep 18 13:30:26.377239 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:26.377199    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6"]
Sep 18 13:30:26.524663 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:26.524628    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svx2k\" (UniqueName: \"kubernetes.io/projected/f768f960-a769-4892-a6e2-366e9c6ff7c7-kube-api-access-svx2k\") pod \"payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6\" (UID: \"f768f960-a769-4892-a6e2-366e9c6ff7c7\") " pod="openshift-ingress/payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6"
Sep 18 13:30:26.524818 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:26.524706    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f768f960-a769-4892-a6e2-366e9c6ff7c7-plugins-config-volume\") pod \"payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6\" (UID: \"f768f960-a769-4892-a6e2-366e9c6ff7c7\") " pod="openshift-ingress/payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6"
Sep 18 13:30:26.625715 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:26.625683    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f768f960-a769-4892-a6e2-366e9c6ff7c7-plugins-config-volume\") pod \"payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6\" (UID: \"f768f960-a769-4892-a6e2-366e9c6ff7c7\") " pod="openshift-ingress/payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6"
Sep 18 13:30:26.625906 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:26.625767    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svx2k\" (UniqueName: \"kubernetes.io/projected/f768f960-a769-4892-a6e2-366e9c6ff7c7-kube-api-access-svx2k\") pod \"payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6\" (UID: \"f768f960-a769-4892-a6e2-366e9c6ff7c7\") " pod="openshift-ingress/payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6"
Sep 18 13:30:26.626422 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:26.626401    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f768f960-a769-4892-a6e2-366e9c6ff7c7-plugins-config-volume\") pod \"payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6\" (UID: \"f768f960-a769-4892-a6e2-366e9c6ff7c7\") " pod="openshift-ingress/payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6"
Sep 18 13:30:26.642326 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:26.642296    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svx2k\" (UniqueName: \"kubernetes.io/projected/f768f960-a769-4892-a6e2-366e9c6ff7c7-kube-api-access-svx2k\") pod \"payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6\" (UID: \"f768f960-a769-4892-a6e2-366e9c6ff7c7\") " pod="openshift-ingress/payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6"
Sep 18 13:30:26.653850 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:26.653818    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6"
Sep 18 13:30:26.958335 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:26.958309    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6"]
Sep 18 13:30:26.960703 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:30:26.960676    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf768f960_a769_4892_a6e2_366e9c6ff7c7.slice/crio-cb9baf360c00bf03bdf1574fe40fa998c0ddef9692135ba77ac8cc4eb202b060 WatchSource:0}: Error finding container cb9baf360c00bf03bdf1574fe40fa998c0ddef9692135ba77ac8cc4eb202b060: Status 404 returned error can't find the container with id cb9baf360c00bf03bdf1574fe40fa998c0ddef9692135ba77ac8cc4eb202b060
Sep 18 13:30:27.705466 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:27.705429    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6" event={"ID":"f768f960-a769-4892-a6e2-366e9c6ff7c7","Type":"ContainerStarted","Data":"bc4c1adfaac5fd88ae5a98eae104a7988ae044c4536ddd2fa3d2443e051f64d2"}
Sep 18 13:30:27.705903 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:27.705486    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6" event={"ID":"f768f960-a769-4892-a6e2-366e9c6ff7c7","Type":"ContainerStarted","Data":"cb9baf360c00bf03bdf1574fe40fa998c0ddef9692135ba77ac8cc4eb202b060"}
Sep 18 13:30:27.705903 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:27.705539    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6"
Sep 18 13:30:27.739755 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:27.739704    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6" podStartSLOduration=1.739681866 podStartE2EDuration="1.739681866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:30:27.73850172 +0000 UTC m=+936.173378124" watchObservedRunningTime="2026-09-18 13:30:27.739681866 +0000 UTC m=+936.174558274"
Sep 18 13:30:38.713282 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:38.713249    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6"
Sep 18 13:30:41.796498 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:41.796462    2566 generic.go:356] "Generic (PLEG): container finished" podID="f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" containerID="f779b1f013aba6bd3cebe8b73c5eb8b0942691780b7642b5bcb48bb23b9ad713" exitCode=0
Sep 18 13:30:41.796944 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:41.796535    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7" event={"ID":"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5","Type":"ContainerDied","Data":"f779b1f013aba6bd3cebe8b73c5eb8b0942691780b7642b5bcb48bb23b9ad713"}
Sep 18 13:30:41.796999 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:41.796977    2566 scope.go:117] "RemoveContainer" containerID="f779b1f013aba6bd3cebe8b73c5eb8b0942691780b7642b5bcb48bb23b9ad713"
Sep 18 13:30:41.798206 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:41.798180    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:30:41.798273 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:41.798247    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:30:41.798310 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:41.798282    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:30:42.803574 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:42.803539    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7" event={"ID":"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5","Type":"ContainerStarted","Data":"dd84d8ed3d0e36a5f4809815440e018e2cf6e23e33d283bb8187651df36a1999"}
Sep 18 13:30:43.311893 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:43.311849    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:43.312122 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:43.312100    2566 patch_prober.go:28] interesting pod/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 13:30:43.312172 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:43.312147    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7" podUID="f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 13:30:44.312855 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:44.312811    2566 patch_prober.go:28] interesting pod/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 13:30:44.313347 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:44.312880    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7" podUID="f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 13:30:45.312798 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:45.312759    2566 patch_prober.go:28] interesting pod/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 13:30:45.312988 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:45.312818    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7" podUID="f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 13:30:46.312389 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:46.312351    2566 patch_prober.go:28] interesting pod/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 13:30:46.312573 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:46.312410    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7" podUID="f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 13:30:47.312689 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:47.312648    2566 patch_prober.go:28] interesting pod/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 13:30:47.313173 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:47.312728    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7" podUID="f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 13:30:48.311910 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:48.311867    2566 patch_prober.go:28] interesting pod/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 13:30:48.312122 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:48.311937    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7" podUID="f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 13:30:49.326794 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:49.326762    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:49.327282 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:49.327169    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:49.327864 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:49.327844    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:30:53.391185 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:53.391149    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92246a6e-7plrd"]
Sep 18 13:30:53.391669 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:53.391654    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f42433c-819f-40ef-a520-df3a5e888160" containerName="istio-proxy"
Sep 18 13:30:53.396247 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:53.396225    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92246a6e-7plrd"
Sep 18 13:30:53.399620 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:53.399585    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:30:53.399737 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:53.399663    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nh2cd\""
Sep 18 13:30:53.404866 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:53.404845    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92246a6e-7plrd"]
Sep 18 13:30:53.492591 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:53.492556    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6aaac10-fb57-42a0-a3d3-c60623d707b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-92246a6e-7plrd\" (UID: \"c6aaac10-fb57-42a0-a3d3-c60623d707b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92246a6e-7plrd"
Sep 18 13:30:53.593306 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:53.593264    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6aaac10-fb57-42a0-a3d3-c60623d707b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-92246a6e-7plrd\" (UID: \"c6aaac10-fb57-42a0-a3d3-c60623d707b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92246a6e-7plrd"
Sep 18 13:30:53.593884 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:53.593864    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6aaac10-fb57-42a0-a3d3-c60623d707b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-92246a6e-7plrd\" (UID: \"c6aaac10-fb57-42a0-a3d3-c60623d707b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92246a6e-7plrd"
Sep 18 13:30:53.708229 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:53.708133    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92246a6e-7plrd"
Sep 18 13:30:53.988797 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:53.988769    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92246a6e-7plrd"]
Sep 18 13:30:53.989151 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:30:53.989121    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6aaac10_fb57_42a0_a3d3_c60623d707b7.slice/crio-84ddba0a0850e8d29c4707bdff6ab8af60921c14aed8a4bc8e12eba6ab2e15ba WatchSource:0}: Error finding container 84ddba0a0850e8d29c4707bdff6ab8af60921c14aed8a4bc8e12eba6ab2e15ba: Status 404 returned error can't find the container with id 84ddba0a0850e8d29c4707bdff6ab8af60921c14aed8a4bc8e12eba6ab2e15ba
Sep 18 13:30:54.868452 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:54.868415    2566 generic.go:356] "Generic (PLEG): container finished" podID="c6aaac10-fb57-42a0-a3d3-c60623d707b7" containerID="0a0a4e429152d9a508a205da253d92368c54e2814827a7f41b9b1264ac6e1d08" exitCode=0
Sep 18 13:30:54.868889 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:54.868504    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92246a6e-7plrd" event={"ID":"c6aaac10-fb57-42a0-a3d3-c60623d707b7","Type":"ContainerDied","Data":"0a0a4e429152d9a508a205da253d92368c54e2814827a7f41b9b1264ac6e1d08"}
Sep 18 13:30:54.868889 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:54.868577    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92246a6e-7plrd" event={"ID":"c6aaac10-fb57-42a0-a3d3-c60623d707b7","Type":"ContainerStarted","Data":"84ddba0a0850e8d29c4707bdff6ab8af60921c14aed8a4bc8e12eba6ab2e15ba"}
Sep 18 13:30:56.013275 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:56.013249    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92246a6e-7plrd"
Sep 18 13:30:56.117472 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:56.117435    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6aaac10-fb57-42a0-a3d3-c60623d707b7-maas-api-service-ca\") pod \"c6aaac10-fb57-42a0-a3d3-c60623d707b7\" (UID: \"c6aaac10-fb57-42a0-a3d3-c60623d707b7\") "
Sep 18 13:30:56.117846 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:56.117812    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6aaac10-fb57-42a0-a3d3-c60623d707b7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c6aaac10-fb57-42a0-a3d3-c60623d707b7" (UID: "c6aaac10-fb57-42a0-a3d3-c60623d707b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:30:56.219211 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:56.219125    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6aaac10-fb57-42a0-a3d3-c60623d707b7-maas-api-service-ca\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:30:56.880080 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:56.880041    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92246a6e-7plrd" event={"ID":"c6aaac10-fb57-42a0-a3d3-c60623d707b7","Type":"ContainerDied","Data":"84ddba0a0850e8d29c4707bdff6ab8af60921c14aed8a4bc8e12eba6ab2e15ba"}
Sep 18 13:30:56.880080 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:56.880071    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92246a6e-7plrd"
Sep 18 13:30:56.880303 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:30:56.880090    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84ddba0a0850e8d29c4707bdff6ab8af60921c14aed8a4bc8e12eba6ab2e15ba"
Sep 18 13:31:01.476572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.476539    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"]
Sep 18 13:31:01.477003 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.476986    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6aaac10-fb57-42a0-a3d3-c60623d707b7" containerName="revoke-keys"
Sep 18 13:31:01.477057 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.477006    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6aaac10-fb57-42a0-a3d3-c60623d707b7" containerName="revoke-keys"
Sep 18 13:31:01.477133 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.477123    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6aaac10-fb57-42a0-a3d3-c60623d707b7" containerName="revoke-keys"
Sep 18 13:31:01.480262 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.480241    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:31:01.482709 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.482688    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-34aeca\"/\"kube-root-ca.crt\""
Sep 18 13:31:01.483344 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.483325    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-34aeca\"/\"openshift-service-ca.crt\""
Sep 18 13:31:01.483344 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.483333    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-34aeca\"/\"default-dockercfg-8wjqr\""
Sep 18 13:31:01.483487 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.483376    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-34aeca\"/\"test-model-34aeca-kserve-self-signed-certs\""
Sep 18 13:31:01.492263 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.492238    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"]
Sep 18 13:31:01.571984 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.571942    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ed58ca-08b3-4da7-b533-0d184bd998fe-tmp-dir\") pod \"test-model-34aeca-kserve-55f79d4f87-c68tg\" (UID: \"65ed58ca-08b3-4da7-b533-0d184bd998fe\") " pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:31:01.572179 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.572017    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ed58ca-08b3-4da7-b533-0d184bd998fe-dshm\") pod \"test-model-34aeca-kserve-55f79d4f87-c68tg\" (UID: \"65ed58ca-08b3-4da7-b533-0d184bd998fe\") " pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:31:01.572179 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.572055    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ed58ca-08b3-4da7-b533-0d184bd998fe-model-cache\") pod \"test-model-34aeca-kserve-55f79d4f87-c68tg\" (UID: \"65ed58ca-08b3-4da7-b533-0d184bd998fe\") " pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:31:01.572179 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.572162    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65ed58ca-08b3-4da7-b533-0d184bd998fe-tls-certs\") pod \"test-model-34aeca-kserve-55f79d4f87-c68tg\" (UID: \"65ed58ca-08b3-4da7-b533-0d184bd998fe\") " pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:31:01.572302 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.572229    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp7j7\" (UniqueName: \"kubernetes.io/projected/65ed58ca-08b3-4da7-b533-0d184bd998fe-kube-api-access-sp7j7\") pod \"test-model-34aeca-kserve-55f79d4f87-c68tg\" (UID: \"65ed58ca-08b3-4da7-b533-0d184bd998fe\") " pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:31:01.572302 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.572292    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65ed58ca-08b3-4da7-b533-0d184bd998fe-home\") pod \"test-model-34aeca-kserve-55f79d4f87-c68tg\" (UID: \"65ed58ca-08b3-4da7-b533-0d184bd998fe\") " pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:31:01.673635 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.673573    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65ed58ca-08b3-4da7-b533-0d184bd998fe-home\") pod \"test-model-34aeca-kserve-55f79d4f87-c68tg\" (UID: \"65ed58ca-08b3-4da7-b533-0d184bd998fe\") " pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:31:01.673844 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.673661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ed58ca-08b3-4da7-b533-0d184bd998fe-tmp-dir\") pod \"test-model-34aeca-kserve-55f79d4f87-c68tg\" (UID: \"65ed58ca-08b3-4da7-b533-0d184bd998fe\") " pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:31:01.673844 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.673706    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ed58ca-08b3-4da7-b533-0d184bd998fe-dshm\") pod \"test-model-34aeca-kserve-55f79d4f87-c68tg\" (UID: \"65ed58ca-08b3-4da7-b533-0d184bd998fe\") " pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:31:01.673844 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.673731    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ed58ca-08b3-4da7-b533-0d184bd998fe-model-cache\") pod \"test-model-34aeca-kserve-55f79d4f87-c68tg\" (UID: \"65ed58ca-08b3-4da7-b533-0d184bd998fe\") " pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:31:01.673844 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.673752    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65ed58ca-08b3-4da7-b533-0d184bd998fe-tls-certs\") pod \"test-model-34aeca-kserve-55f79d4f87-c68tg\" (UID: \"65ed58ca-08b3-4da7-b533-0d184bd998fe\") " pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:31:01.673844 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.673833    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp7j7\" (UniqueName: \"kubernetes.io/projected/65ed58ca-08b3-4da7-b533-0d184bd998fe-kube-api-access-sp7j7\") pod \"test-model-34aeca-kserve-55f79d4f87-c68tg\" (UID: \"65ed58ca-08b3-4da7-b533-0d184bd998fe\") " pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:31:01.674154 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.674068    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65ed58ca-08b3-4da7-b533-0d184bd998fe-home\") pod \"test-model-34aeca-kserve-55f79d4f87-c68tg\" (UID: \"65ed58ca-08b3-4da7-b533-0d184bd998fe\") " pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:31:01.674207 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.674149    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ed58ca-08b3-4da7-b533-0d184bd998fe-tmp-dir\") pod \"test-model-34aeca-kserve-55f79d4f87-c68tg\" (UID: \"65ed58ca-08b3-4da7-b533-0d184bd998fe\") " pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:31:01.674207 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.674175    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ed58ca-08b3-4da7-b533-0d184bd998fe-model-cache\") pod \"test-model-34aeca-kserve-55f79d4f87-c68tg\" (UID: \"65ed58ca-08b3-4da7-b533-0d184bd998fe\") " pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:31:01.676015 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.675978    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ed58ca-08b3-4da7-b533-0d184bd998fe-dshm\") pod \"test-model-34aeca-kserve-55f79d4f87-c68tg\" (UID: \"65ed58ca-08b3-4da7-b533-0d184bd998fe\") " pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:31:01.676509 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.676487    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65ed58ca-08b3-4da7-b533-0d184bd998fe-tls-certs\") pod \"test-model-34aeca-kserve-55f79d4f87-c68tg\" (UID: \"65ed58ca-08b3-4da7-b533-0d184bd998fe\") " pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:31:01.682914 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.682892    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp7j7\" (UniqueName: \"kubernetes.io/projected/65ed58ca-08b3-4da7-b533-0d184bd998fe-kube-api-access-sp7j7\") pod \"test-model-34aeca-kserve-55f79d4f87-c68tg\" (UID: \"65ed58ca-08b3-4da7-b533-0d184bd998fe\") " pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:31:01.790436 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:01.790333    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:31:02.066114 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:02.066088    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"]
Sep 18 13:31:02.068590 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:31:02.068561    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65ed58ca_08b3_4da7_b533_0d184bd998fe.slice/crio-885eb6e21f990e7d27dcdb03bb47dbd569bb2eba2eeb4b62959d15db246f0bb1 WatchSource:0}: Error finding container 885eb6e21f990e7d27dcdb03bb47dbd569bb2eba2eeb4b62959d15db246f0bb1: Status 404 returned error can't find the container with id 885eb6e21f990e7d27dcdb03bb47dbd569bb2eba2eeb4b62959d15db246f0bb1
Sep 18 13:31:02.918272 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:02.918228    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg" event={"ID":"65ed58ca-08b3-4da7-b533-0d184bd998fe","Type":"ContainerStarted","Data":"885eb6e21f990e7d27dcdb03bb47dbd569bb2eba2eeb4b62959d15db246f0bb1"}
Sep 18 13:31:03.925479 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:03.925440    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg" event={"ID":"65ed58ca-08b3-4da7-b533-0d184bd998fe","Type":"ContainerStarted","Data":"a178ca5fb631b3edd72993b8a96edb019554f95362cb5aa74d966ce28d009ebf"}
Sep 18 13:31:03.942940 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:03.942896    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg" podStartSLOduration=1.860227903 podStartE2EDuration="2.942881769s" totalImagesPullingTime="1.082653866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:31:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:31:03.941559089 +0000 UTC m=+972.376435494" watchObservedRunningTime="2026-09-18 13:31:03.942881769 +0000 UTC m=+972.377758173"
Sep 18 13:31:04.613289 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:04.613257    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6"]
Sep 18 13:31:04.613478 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:04.613457    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6" podUID="f768f960-a769-4892-a6e2-366e9c6ff7c7" containerName="payload-processing" containerID="cri-o://bc4c1adfaac5fd88ae5a98eae104a7988ae044c4536ddd2fa3d2443e051f64d2" gracePeriod=30
Sep 18 13:31:04.873864 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:04.873805    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6"
Sep 18 13:31:04.946861 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:04.946818    2566 generic.go:356] "Generic (PLEG): container finished" podID="f768f960-a769-4892-a6e2-366e9c6ff7c7" containerID="bc4c1adfaac5fd88ae5a98eae104a7988ae044c4536ddd2fa3d2443e051f64d2" exitCode=0
Sep 18 13:31:04.948372 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:04.947650    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6"
Sep 18 13:31:04.948372 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:04.947957    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6" event={"ID":"f768f960-a769-4892-a6e2-366e9c6ff7c7","Type":"ContainerDied","Data":"bc4c1adfaac5fd88ae5a98eae104a7988ae044c4536ddd2fa3d2443e051f64d2"}
Sep 18 13:31:04.948372 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:04.948007    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6" event={"ID":"f768f960-a769-4892-a6e2-366e9c6ff7c7","Type":"ContainerDied","Data":"cb9baf360c00bf03bdf1574fe40fa998c0ddef9692135ba77ac8cc4eb202b060"}
Sep 18 13:31:04.948372 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:04.948037    2566 scope.go:117] "RemoveContainer" containerID="bc4c1adfaac5fd88ae5a98eae104a7988ae044c4536ddd2fa3d2443e051f64d2"
Sep 18 13:31:04.970416 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:04.970379    2566 scope.go:117] "RemoveContainer" containerID="bc4c1adfaac5fd88ae5a98eae104a7988ae044c4536ddd2fa3d2443e051f64d2"
Sep 18 13:31:04.971703 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:31:04.971667    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc4c1adfaac5fd88ae5a98eae104a7988ae044c4536ddd2fa3d2443e051f64d2\": container with ID starting with bc4c1adfaac5fd88ae5a98eae104a7988ae044c4536ddd2fa3d2443e051f64d2 not found: ID does not exist" containerID="bc4c1adfaac5fd88ae5a98eae104a7988ae044c4536ddd2fa3d2443e051f64d2"
Sep 18 13:31:04.971902 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:04.971874    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc4c1adfaac5fd88ae5a98eae104a7988ae044c4536ddd2fa3d2443e051f64d2"} err="failed to get container status \"bc4c1adfaac5fd88ae5a98eae104a7988ae044c4536ddd2fa3d2443e051f64d2\": rpc error: code = NotFound desc = could not find container \"bc4c1adfaac5fd88ae5a98eae104a7988ae044c4536ddd2fa3d2443e051f64d2\": container with ID starting with bc4c1adfaac5fd88ae5a98eae104a7988ae044c4536ddd2fa3d2443e051f64d2 not found: ID does not exist"
Sep 18 13:31:05.007280 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:05.007241    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f768f960-a769-4892-a6e2-366e9c6ff7c7-plugins-config-volume\") pod \"f768f960-a769-4892-a6e2-366e9c6ff7c7\" (UID: \"f768f960-a769-4892-a6e2-366e9c6ff7c7\") "
Sep 18 13:31:05.007432 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:05.007397    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svx2k\" (UniqueName: \"kubernetes.io/projected/f768f960-a769-4892-a6e2-366e9c6ff7c7-kube-api-access-svx2k\") pod \"f768f960-a769-4892-a6e2-366e9c6ff7c7\" (UID: \"f768f960-a769-4892-a6e2-366e9c6ff7c7\") "
Sep 18 13:31:05.007645 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:05.007610    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f768f960-a769-4892-a6e2-366e9c6ff7c7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f768f960-a769-4892-a6e2-366e9c6ff7c7" (UID: "f768f960-a769-4892-a6e2-366e9c6ff7c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:31:05.007832 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:05.007809    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f768f960-a769-4892-a6e2-366e9c6ff7c7-plugins-config-volume\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:05.009626 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:05.009583    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f768f960-a769-4892-a6e2-366e9c6ff7c7-kube-api-access-svx2k" (OuterVolumeSpecName: "kube-api-access-svx2k") pod "f768f960-a769-4892-a6e2-366e9c6ff7c7" (UID: "f768f960-a769-4892-a6e2-366e9c6ff7c7"). InnerVolumeSpecName "kube-api-access-svx2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:31:05.012847 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:05.012823    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92246a6e-7plrd"]
Sep 18 13:31:05.016993 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:05.016968    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-92246a6e-7plrd"]
Sep 18 13:31:05.109060 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:05.109025    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svx2k\" (UniqueName: \"kubernetes.io/projected/f768f960-a769-4892-a6e2-366e9c6ff7c7-kube-api-access-svx2k\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:05.271933 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:05.271896    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6"]
Sep 18 13:31:05.273979 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:05.273957    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-92246a6e-7987dbf77c-8khm6"]
Sep 18 13:31:06.077167 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:06.077134    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6aaac10-fb57-42a0-a3d3-c60623d707b7" path="/var/lib/kubelet/pods/c6aaac10-fb57-42a0-a3d3-c60623d707b7/volumes"
Sep 18 13:31:06.077531 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:06.077517    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f768f960-a769-4892-a6e2-366e9c6ff7c7" path="/var/lib/kubelet/pods/f768f960-a769-4892-a6e2-366e9c6ff7c7/volumes"
Sep 18 13:31:06.270957 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:06.270925    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"]
Sep 18 13:31:06.271178 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:06.271155    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7" podUID="f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" containerName="istio-proxy" containerID="cri-o://dd84d8ed3d0e36a5f4809815440e018e2cf6e23e33d283bb8187651df36a1999" gracePeriod=30
Sep 18 13:31:07.034992 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.034950    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"]
Sep 18 13:31:07.035355 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.035342    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f768f960-a769-4892-a6e2-366e9c6ff7c7" containerName="payload-processing"
Sep 18 13:31:07.035399 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.035357    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f768f960-a769-4892-a6e2-366e9c6ff7c7" containerName="payload-processing"
Sep 18 13:31:07.035437 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.035430    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f768f960-a769-4892-a6e2-366e9c6ff7c7" containerName="payload-processing"
Sep 18 13:31:07.039708 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.039690    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.042050 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.042031    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-cc741da4-openshift-default-dockercfg-8mm95\""
Sep 18 13:31:07.056699 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.056673    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"]
Sep 18 13:31:07.129236 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.129198    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istio-podinfo\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.129236 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.129241    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istio-envoy\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.129724 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.129266    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-credential-socket\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.129724 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.129341    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stw4v\" (UniqueName: \"kubernetes.io/projected/d24e2aab-e800-4304-9fab-f66e9f5f11aa-kube-api-access-stw4v\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.129724 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.129401    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istio-token\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.129724 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.129443    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istiod-ca-cert\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.129724 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.129507    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-workload-socket\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.129724 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.129565    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-workload-certs\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.129724 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.129631    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istio-data\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.231120 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.231075    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istiod-ca-cert\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.231313 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.231174    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-workload-socket\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.231313 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.231252    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-workload-certs\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.231313 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.231301    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istio-data\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.231486 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.231383    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istio-podinfo\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.231486 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.231418    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istio-envoy\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.231486 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.231462    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-credential-socket\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.231656 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.231533    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stw4v\" (UniqueName: \"kubernetes.io/projected/d24e2aab-e800-4304-9fab-f66e9f5f11aa-kube-api-access-stw4v\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.231656 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.231569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istio-token\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.232085 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.232058    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istiod-ca-cert\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.232544 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.232519    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-workload-socket\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.232942 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.232899    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-workload-certs\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.233366 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.233346    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istio-data\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.233878 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.233857    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-credential-socket\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.235363 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.235318    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istio-envoy\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.236559 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.236518    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istio-podinfo\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.245707 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.245652    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stw4v\" (UniqueName: \"kubernetes.io/projected/d24e2aab-e800-4304-9fab-f66e9f5f11aa-kube-api-access-stw4v\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.246625 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.246509    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istio-token\") pod \"e2e-derive-cc741da4-openshift-default-7f57475948-48fvs\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") " pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.350576 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.350541    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:07.652758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.652669    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"]
Sep 18 13:31:07.661563 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:31:07.661535    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd24e2aab_e800_4304_9fab_f66e9f5f11aa.slice/crio-948ec587eb502884eb2c90dec85dcfb5e66bd00f7b5168dd77a6ef97e5f9d2c5 WatchSource:0}: Error finding container 948ec587eb502884eb2c90dec85dcfb5e66bd00f7b5168dd77a6ef97e5f9d2c5: Status 404 returned error can't find the container with id 948ec587eb502884eb2c90dec85dcfb5e66bd00f7b5168dd77a6ef97e5f9d2c5
Sep 18 13:31:07.664491 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.664457    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:31:07.664586 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.664529    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:31:07.664586 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.664556    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:31:07.965657 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.965543    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs" event={"ID":"d24e2aab-e800-4304-9fab-f66e9f5f11aa","Type":"ContainerStarted","Data":"7dae7ad851dba728b0655b9182b0a31301e76181d1e3f07a396e57cbb23e4dc4"}
Sep 18 13:31:07.965657 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.965624    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs" event={"ID":"d24e2aab-e800-4304-9fab-f66e9f5f11aa","Type":"ContainerStarted","Data":"948ec587eb502884eb2c90dec85dcfb5e66bd00f7b5168dd77a6ef97e5f9d2c5"}
Sep 18 13:31:07.985863 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:07.985812    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs" podStartSLOduration=0.985796383 podStartE2EDuration="985.796383ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:31:07.983911739 +0000 UTC m=+976.418788138" watchObservedRunningTime="2026-09-18 13:31:07.985796383 +0000 UTC m=+976.420672787"
Sep 18 13:31:08.351350 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:08.351319    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:08.357314 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:08.357288    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:08.972246 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:08.972215    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:08.973267 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:08.973245    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:11.563560 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.563523    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:31:11.679453 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.679415    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-workload-certs\") pod \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") "
Sep 18 13:31:11.679453 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.679467    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-credential-socket\") pod \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") "
Sep 18 13:31:11.679730 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.679492    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwdvn\" (UniqueName: \"kubernetes.io/projected/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-kube-api-access-lwdvn\") pod \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") "
Sep 18 13:31:11.679730 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.679514    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istio-podinfo\") pod \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") "
Sep 18 13:31:11.679730 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.679534    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istio-token\") pod \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") "
Sep 18 13:31:11.679730 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.679560    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istio-data\") pod \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") "
Sep 18 13:31:11.679730 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.679644    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istio-envoy\") pod \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") "
Sep 18 13:31:11.679730 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.679687    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-workload-socket\") pod \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") "
Sep 18 13:31:11.679730 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.679727    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istiod-ca-cert\") pod \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\" (UID: \"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5\") "
Sep 18 13:31:11.680096 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.679744    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" (UID: "f1a1789e-8e58-406f-8d4d-bf2528ca5cd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:31:11.680096 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.679783    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" (UID: "f1a1789e-8e58-406f-8d4d-bf2528ca5cd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:31:11.680096 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.679948    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" (UID: "f1a1789e-8e58-406f-8d4d-bf2528ca5cd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:31:11.680253 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.680119    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istio-data" (OuterVolumeSpecName: "istio-data") pod "f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" (UID: "f1a1789e-8e58-406f-8d4d-bf2528ca5cd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:31:11.680253 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.680125    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" (UID: "f1a1789e-8e58-406f-8d4d-bf2528ca5cd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:31:11.680253 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.680148    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-workload-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:11.680253 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.680171    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-workload-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:11.680253 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.680185    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-credential-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:11.681947 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.681919    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istio-token" (OuterVolumeSpecName: "istio-token") pod "f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" (UID: "f1a1789e-8e58-406f-8d4d-bf2528ca5cd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:31:11.682063 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.682044    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" (UID: "f1a1789e-8e58-406f-8d4d-bf2528ca5cd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 13:31:11.682135 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.682112    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" (UID: "f1a1789e-8e58-406f-8d4d-bf2528ca5cd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:31:11.682185 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.682142    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-kube-api-access-lwdvn" (OuterVolumeSpecName: "kube-api-access-lwdvn") pod "f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" (UID: "f1a1789e-8e58-406f-8d4d-bf2528ca5cd5"). InnerVolumeSpecName "kube-api-access-lwdvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:31:11.781728 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.781690    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwdvn\" (UniqueName: \"kubernetes.io/projected/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-kube-api-access-lwdvn\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:11.781728 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.781721    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istio-podinfo\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:11.781728 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.781731    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istio-token\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:11.781728 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.781739    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istio-data\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:11.782000 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.781751    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istio-envoy\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:11.782000 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.781758    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5-istiod-ca-cert\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:11.790543 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.790514    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:31:11.790688 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.790556    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:31:11.803453 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.803370    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:31:11.990710 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.990671    2566 generic.go:356] "Generic (PLEG): container finished" podID="f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" containerID="dd84d8ed3d0e36a5f4809815440e018e2cf6e23e33d283bb8187651df36a1999" exitCode=0
Sep 18 13:31:11.990904 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.990726    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"
Sep 18 13:31:11.990904 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.990750    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7" event={"ID":"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5","Type":"ContainerDied","Data":"dd84d8ed3d0e36a5f4809815440e018e2cf6e23e33d283bb8187651df36a1999"}
Sep 18 13:31:11.990904 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.990795    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7" event={"ID":"f1a1789e-8e58-406f-8d4d-bf2528ca5cd5","Type":"ContainerDied","Data":"20cceadab7946f268d403c4cb65a2966721e67b2bd518622796e0fc32437867a"}
Sep 18 13:31:11.990904 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:11.990818    2566 scope.go:117] "RemoveContainer" containerID="dd84d8ed3d0e36a5f4809815440e018e2cf6e23e33d283bb8187651df36a1999"
Sep 18 13:31:12.002779 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:12.002752    2566 scope.go:117] "RemoveContainer" containerID="f779b1f013aba6bd3cebe8b73c5eb8b0942691780b7642b5bcb48bb23b9ad713"
Sep 18 13:31:12.004381 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:12.004359    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:31:12.012869 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:12.012852    2566 scope.go:117] "RemoveContainer" containerID="dd84d8ed3d0e36a5f4809815440e018e2cf6e23e33d283bb8187651df36a1999"
Sep 18 13:31:12.013188 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:31:12.013170    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd84d8ed3d0e36a5f4809815440e018e2cf6e23e33d283bb8187651df36a1999\": container with ID starting with dd84d8ed3d0e36a5f4809815440e018e2cf6e23e33d283bb8187651df36a1999 not found: ID does not exist" containerID="dd84d8ed3d0e36a5f4809815440e018e2cf6e23e33d283bb8187651df36a1999"
Sep 18 13:31:12.013239 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:12.013198    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd84d8ed3d0e36a5f4809815440e018e2cf6e23e33d283bb8187651df36a1999"} err="failed to get container status \"dd84d8ed3d0e36a5f4809815440e018e2cf6e23e33d283bb8187651df36a1999\": rpc error: code = NotFound desc = could not find container \"dd84d8ed3d0e36a5f4809815440e018e2cf6e23e33d283bb8187651df36a1999\": container with ID starting with dd84d8ed3d0e36a5f4809815440e018e2cf6e23e33d283bb8187651df36a1999 not found: ID does not exist"
Sep 18 13:31:12.013239 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:12.013216    2566 scope.go:117] "RemoveContainer" containerID="f779b1f013aba6bd3cebe8b73c5eb8b0942691780b7642b5bcb48bb23b9ad713"
Sep 18 13:31:12.013475 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:31:12.013453    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f779b1f013aba6bd3cebe8b73c5eb8b0942691780b7642b5bcb48bb23b9ad713\": container with ID starting with f779b1f013aba6bd3cebe8b73c5eb8b0942691780b7642b5bcb48bb23b9ad713 not found: ID does not exist" containerID="f779b1f013aba6bd3cebe8b73c5eb8b0942691780b7642b5bcb48bb23b9ad713"
Sep 18 13:31:12.013524 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:12.013481    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f779b1f013aba6bd3cebe8b73c5eb8b0942691780b7642b5bcb48bb23b9ad713"} err="failed to get container status \"f779b1f013aba6bd3cebe8b73c5eb8b0942691780b7642b5bcb48bb23b9ad713\": rpc error: code = NotFound desc = could not find container \"f779b1f013aba6bd3cebe8b73c5eb8b0942691780b7642b5bcb48bb23b9ad713\": container with ID starting with f779b1f013aba6bd3cebe8b73c5eb8b0942691780b7642b5bcb48bb23b9ad713 not found: ID does not exist"
Sep 18 13:31:12.026372 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:12.026345    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"]
Sep 18 13:31:12.038547 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:12.038521    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-92246a6e-openshift-default-67c76c746b-fw9v7"]
Sep 18 13:31:12.078900 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:12.078818    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" path="/var/lib/kubelet/pods/f1a1789e-8e58-406f-8d4d-bf2528ca5cd5/volumes"
Sep 18 13:31:23.524437 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:23.524393    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b7292e-skt2k"]
Sep 18 13:31:23.525041 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:23.525017    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" containerName="istio-proxy"
Sep 18 13:31:23.525130 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:23.525044    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" containerName="istio-proxy"
Sep 18 13:31:23.525130 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:23.525063    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" containerName="istio-proxy"
Sep 18 13:31:23.525130 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:23.525071    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" containerName="istio-proxy"
Sep 18 13:31:23.525277 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:23.525175    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" containerName="istio-proxy"
Sep 18 13:31:23.525277 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:23.525194    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1a1789e-8e58-406f-8d4d-bf2528ca5cd5" containerName="istio-proxy"
Sep 18 13:31:23.528046 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:23.528024    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b7292e-skt2k"
Sep 18 13:31:23.535949 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:23.535928    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:31:23.536452 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:23.536435    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nh2cd\""
Sep 18 13:31:23.562548 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:23.562515    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b7292e-skt2k"]
Sep 18 13:31:23.576356 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:23.576324    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45192efd-a3ec-49d6-a931-2dfd69dc0482-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b7292e-skt2k\" (UID: \"45192efd-a3ec-49d6-a931-2dfd69dc0482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b7292e-skt2k"
Sep 18 13:31:23.677846 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:23.677814    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45192efd-a3ec-49d6-a931-2dfd69dc0482-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b7292e-skt2k\" (UID: \"45192efd-a3ec-49d6-a931-2dfd69dc0482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b7292e-skt2k"
Sep 18 13:31:23.678692 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:23.678668    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45192efd-a3ec-49d6-a931-2dfd69dc0482-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b7292e-skt2k\" (UID: \"45192efd-a3ec-49d6-a931-2dfd69dc0482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b7292e-skt2k"
Sep 18 13:31:23.838922 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:23.838840    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b7292e-skt2k"
Sep 18 13:31:24.115504 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:24.115471    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b7292e-skt2k"]
Sep 18 13:31:24.118745 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:31:24.118719    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45192efd_a3ec_49d6_a931_2dfd69dc0482.slice/crio-d4b7723dc12e5750e9f8e4289531e7f5d9b0277fa8c6dc6bb56a6d6b20bc9c3c WatchSource:0}: Error finding container d4b7723dc12e5750e9f8e4289531e7f5d9b0277fa8c6dc6bb56a6d6b20bc9c3c: Status 404 returned error can't find the container with id d4b7723dc12e5750e9f8e4289531e7f5d9b0277fa8c6dc6bb56a6d6b20bc9c3c
Sep 18 13:31:25.063419 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:25.063382    2566 generic.go:356] "Generic (PLEG): container finished" podID="45192efd-a3ec-49d6-a931-2dfd69dc0482" containerID="2112ebc900f496725ba3c30b39c0161ea895792d228f37471bac3fbc4373df04" exitCode=0
Sep 18 13:31:25.063883 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:25.063467    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b7292e-skt2k" event={"ID":"45192efd-a3ec-49d6-a931-2dfd69dc0482","Type":"ContainerDied","Data":"2112ebc900f496725ba3c30b39c0161ea895792d228f37471bac3fbc4373df04"}
Sep 18 13:31:25.063883 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:25.063521    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b7292e-skt2k" event={"ID":"45192efd-a3ec-49d6-a931-2dfd69dc0482","Type":"ContainerStarted","Data":"d4b7723dc12e5750e9f8e4289531e7f5d9b0277fa8c6dc6bb56a6d6b20bc9c3c"}
Sep 18 13:31:26.219301 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:26.219278    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b7292e-skt2k"
Sep 18 13:31:26.301411 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:26.301361    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45192efd-a3ec-49d6-a931-2dfd69dc0482-maas-api-service-ca\") pod \"45192efd-a3ec-49d6-a931-2dfd69dc0482\" (UID: \"45192efd-a3ec-49d6-a931-2dfd69dc0482\") "
Sep 18 13:31:26.301736 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:26.301712    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45192efd-a3ec-49d6-a931-2dfd69dc0482-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "45192efd-a3ec-49d6-a931-2dfd69dc0482" (UID: "45192efd-a3ec-49d6-a931-2dfd69dc0482"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:31:26.402810 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:26.402766    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45192efd-a3ec-49d6-a931-2dfd69dc0482-maas-api-service-ca\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:27.075443 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:27.075412    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b7292e-skt2k"
Sep 18 13:31:27.075665 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:27.075453    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b7292e-skt2k" event={"ID":"45192efd-a3ec-49d6-a931-2dfd69dc0482","Type":"ContainerDied","Data":"d4b7723dc12e5750e9f8e4289531e7f5d9b0277fa8c6dc6bb56a6d6b20bc9c3c"}
Sep 18 13:31:27.075665 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:27.075494    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4b7723dc12e5750e9f8e4289531e7f5d9b0277fa8c6dc6bb56a6d6b20bc9c3c"
Sep 18 13:31:34.117558 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:34.117525    2566 generic.go:356] "Generic (PLEG): container finished" podID="d24e2aab-e800-4304-9fab-f66e9f5f11aa" containerID="7dae7ad851dba728b0655b9182b0a31301e76181d1e3f07a396e57cbb23e4dc4" exitCode=0
Sep 18 13:31:34.118046 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:34.117624    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs" event={"ID":"d24e2aab-e800-4304-9fab-f66e9f5f11aa","Type":"ContainerDied","Data":"7dae7ad851dba728b0655b9182b0a31301e76181d1e3f07a396e57cbb23e4dc4"}
Sep 18 13:31:34.118114 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:34.118063    2566 scope.go:117] "RemoveContainer" containerID="7dae7ad851dba728b0655b9182b0a31301e76181d1e3f07a396e57cbb23e4dc4"
Sep 18 13:31:34.119394 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:34.119363    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:31:34.119488 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:34.119432    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:31:34.119488 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:34.119463    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:31:34.354374 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:34.354330    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cc741da4-h5xcd"]
Sep 18 13:31:34.354784 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:34.354767    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45192efd-a3ec-49d6-a931-2dfd69dc0482" containerName="revoke-keys"
Sep 18 13:31:34.354784 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:34.354784    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="45192efd-a3ec-49d6-a931-2dfd69dc0482" containerName="revoke-keys"
Sep 18 13:31:34.354959 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:34.354864    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="45192efd-a3ec-49d6-a931-2dfd69dc0482" containerName="revoke-keys"
Sep 18 13:31:34.357005 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:34.356983    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cc741da4-h5xcd"
Sep 18 13:31:34.361960 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:34.361938    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nh2cd\""
Sep 18 13:31:34.362074 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:34.361994    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:31:34.371673 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:34.371646    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cc741da4-h5xcd"]
Sep 18 13:31:34.489794 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:34.489757    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55404ea4-fcd5-4371-ba3e-fa3a8c3929fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-cc741da4-h5xcd\" (UID: \"55404ea4-fcd5-4371-ba3e-fa3a8c3929fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cc741da4-h5xcd"
Sep 18 13:31:34.590770 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:34.590729    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55404ea4-fcd5-4371-ba3e-fa3a8c3929fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-cc741da4-h5xcd\" (UID: \"55404ea4-fcd5-4371-ba3e-fa3a8c3929fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cc741da4-h5xcd"
Sep 18 13:31:34.591391 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:34.591371    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55404ea4-fcd5-4371-ba3e-fa3a8c3929fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-cc741da4-h5xcd\" (UID: \"55404ea4-fcd5-4371-ba3e-fa3a8c3929fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cc741da4-h5xcd"
Sep 18 13:31:34.668301 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:34.668213    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cc741da4-h5xcd"
Sep 18 13:31:34.947226 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:34.942172    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cc741da4-h5xcd"]
Sep 18 13:31:34.947484 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:31:34.947451    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55404ea4_fcd5_4371_ba3e_fa3a8c3929fa.slice/crio-5b4087f9a53fbdc832482e4629a8ca815feeddadccdfd1939eec6dfa2f8bce7f WatchSource:0}: Error finding container 5b4087f9a53fbdc832482e4629a8ca815feeddadccdfd1939eec6dfa2f8bce7f: Status 404 returned error can't find the container with id 5b4087f9a53fbdc832482e4629a8ca815feeddadccdfd1939eec6dfa2f8bce7f
Sep 18 13:31:35.124934 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:35.124887    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cc741da4-h5xcd" event={"ID":"55404ea4-fcd5-4371-ba3e-fa3a8c3929fa","Type":"ContainerStarted","Data":"5b4087f9a53fbdc832482e4629a8ca815feeddadccdfd1939eec6dfa2f8bce7f"}
Sep 18 13:31:35.126880 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:35.126854    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs" event={"ID":"d24e2aab-e800-4304-9fab-f66e9f5f11aa","Type":"ContainerStarted","Data":"9845b40193c2ff50f59474ba61c0d6ecb4d6481580ff558a29e6790195f2e45a"}
Sep 18 13:31:35.351168 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:35.351126    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:35.352657 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:35.352586    2566 patch_prober.go:28] interesting pod/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 18 13:31:35.352811 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:35.352704    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs" podUID="d24e2aab-e800-4304-9fab-f66e9f5f11aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 13:31:36.136826 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:36.136785    2566 generic.go:356] "Generic (PLEG): container finished" podID="55404ea4-fcd5-4371-ba3e-fa3a8c3929fa" containerID="e2756b62618944cdb428c37b2a9a2fb04252fb80059cdf2dff6cd528a95ff37c" exitCode=0
Sep 18 13:31:36.138672 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:36.138643    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cc741da4-h5xcd" event={"ID":"55404ea4-fcd5-4371-ba3e-fa3a8c3929fa","Type":"ContainerDied","Data":"e2756b62618944cdb428c37b2a9a2fb04252fb80059cdf2dff6cd528a95ff37c"}
Sep 18 13:31:36.351990 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:36.351949    2566 patch_prober.go:28] interesting pod/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 18 13:31:36.352177 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:36.352034    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs" podUID="d24e2aab-e800-4304-9fab-f66e9f5f11aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 13:31:37.326362 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:37.326337    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cc741da4-h5xcd"
Sep 18 13:31:37.351016 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:37.350979    2566 patch_prober.go:28] interesting pod/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 18 13:31:37.352184 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:37.352144    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs" podUID="d24e2aab-e800-4304-9fab-f66e9f5f11aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 13:31:37.417422 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:37.417331    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55404ea4-fcd5-4371-ba3e-fa3a8c3929fa-maas-api-service-ca\") pod \"55404ea4-fcd5-4371-ba3e-fa3a8c3929fa\" (UID: \"55404ea4-fcd5-4371-ba3e-fa3a8c3929fa\") "
Sep 18 13:31:37.417687 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:37.417664    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55404ea4-fcd5-4371-ba3e-fa3a8c3929fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "55404ea4-fcd5-4371-ba3e-fa3a8c3929fa" (UID: "55404ea4-fcd5-4371-ba3e-fa3a8c3929fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:31:37.518401 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:37.518365    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55404ea4-fcd5-4371-ba3e-fa3a8c3929fa-maas-api-service-ca\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:38.156078 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:38.156033    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cc741da4-h5xcd" event={"ID":"55404ea4-fcd5-4371-ba3e-fa3a8c3929fa","Type":"ContainerDied","Data":"5b4087f9a53fbdc832482e4629a8ca815feeddadccdfd1939eec6dfa2f8bce7f"}
Sep 18 13:31:38.156287 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:38.156102    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b4087f9a53fbdc832482e4629a8ca815feeddadccdfd1939eec6dfa2f8bce7f"
Sep 18 13:31:38.156287 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:38.156172    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cc741da4-h5xcd"
Sep 18 13:31:38.360619 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:38.360005    2566 patch_prober.go:28] interesting pod/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 18 13:31:38.360619 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:38.360090    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs" podUID="d24e2aab-e800-4304-9fab-f66e9f5f11aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 13:31:38.850696 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:38.850648    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b7292e-skt2k"]
Sep 18 13:31:38.854815 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:38.854779    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b7292e-skt2k"]
Sep 18 13:31:38.972978 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:38.972931    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:39.352381 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:39.351913    2566 patch_prober.go:28] interesting pod/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 18 13:31:39.352381 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:39.351994    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs" podUID="d24e2aab-e800-4304-9fab-f66e9f5f11aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 13:31:40.076878 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:40.076846    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45192efd-a3ec-49d6-a931-2dfd69dc0482" path="/var/lib/kubelet/pods/45192efd-a3ec-49d6-a931-2dfd69dc0482/volumes"
Sep 18 13:31:40.351342 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:40.351301    2566 patch_prober.go:28] interesting pod/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 18 13:31:40.351648 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:40.351615    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs" podUID="d24e2aab-e800-4304-9fab-f66e9f5f11aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 13:31:41.355043 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:41.355010    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:41.356061 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:41.356037    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:44.852405 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:44.852370    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cc741da4-h5xcd"]
Sep 18 13:31:44.858798 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:44.858772    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cc741da4-h5xcd"]
Sep 18 13:31:45.041079 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:45.041046    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"]
Sep 18 13:31:45.041259 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:45.041225    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs" podUID="d24e2aab-e800-4304-9fab-f66e9f5f11aa" containerName="istio-proxy" containerID="cri-o://9845b40193c2ff50f59474ba61c0d6ecb4d6481580ff558a29e6790195f2e45a" gracePeriod=30
Sep 18 13:31:46.077294 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:46.077258    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55404ea4-fcd5-4371-ba3e-fa3a8c3929fa" path="/var/lib/kubelet/pods/55404ea4-fcd5-4371-ba3e-fa3a8c3929fa/volumes"
Sep 18 13:31:49.062635 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:49.062584    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"]
Sep 18 13:31:49.063093 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:49.062808    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc" podUID="1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" containerName="istio-proxy" containerID="cri-o://d87c83bdb08e2090753e89efac5e6ad0ce01de2b6b6f3495e3d32191f389bb15" gracePeriod=30
Sep 18 13:31:50.246374 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.246343    2566 generic.go:356] "Generic (PLEG): container finished" podID="d24e2aab-e800-4304-9fab-f66e9f5f11aa" containerID="9845b40193c2ff50f59474ba61c0d6ecb4d6481580ff558a29e6790195f2e45a" exitCode=0
Sep 18 13:31:50.246839 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.246391    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs" event={"ID":"d24e2aab-e800-4304-9fab-f66e9f5f11aa","Type":"ContainerDied","Data":"9845b40193c2ff50f59474ba61c0d6ecb4d6481580ff558a29e6790195f2e45a"}
Sep 18 13:31:50.246839 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.246446    2566 scope.go:117] "RemoveContainer" containerID="7dae7ad851dba728b0655b9182b0a31301e76181d1e3f07a396e57cbb23e4dc4"
Sep 18 13:31:50.305304 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.305272    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:50.459686 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.459578    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-workload-certs\") pod \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") "
Sep 18 13:31:50.459686 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.459652    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istio-token\") pod \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") "
Sep 18 13:31:50.459686 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.459676    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istiod-ca-cert\") pod \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") "
Sep 18 13:31:50.459976 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.459695    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istio-data\") pod \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") "
Sep 18 13:31:50.459976 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.459716    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-credential-socket\") pod \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") "
Sep 18 13:31:50.459976 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.459766    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istio-podinfo\") pod \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") "
Sep 18 13:31:50.459976 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.459811    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-workload-socket\") pod \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") "
Sep 18 13:31:50.459976 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.459850    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istio-envoy\") pod \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") "
Sep 18 13:31:50.459976 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.459880    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stw4v\" (UniqueName: \"kubernetes.io/projected/d24e2aab-e800-4304-9fab-f66e9f5f11aa-kube-api-access-stw4v\") pod \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\" (UID: \"d24e2aab-e800-4304-9fab-f66e9f5f11aa\") "
Sep 18 13:31:50.459976 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.459934    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d24e2aab-e800-4304-9fab-f66e9f5f11aa" (UID: "d24e2aab-e800-4304-9fab-f66e9f5f11aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:31:50.460339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.460043    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istio-data" (OuterVolumeSpecName: "istio-data") pod "d24e2aab-e800-4304-9fab-f66e9f5f11aa" (UID: "d24e2aab-e800-4304-9fab-f66e9f5f11aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:31:50.460339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.460105    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d24e2aab-e800-4304-9fab-f66e9f5f11aa" (UID: "d24e2aab-e800-4304-9fab-f66e9f5f11aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:31:50.460339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.460138    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d24e2aab-e800-4304-9fab-f66e9f5f11aa" (UID: "d24e2aab-e800-4304-9fab-f66e9f5f11aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:31:50.460339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.460310    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-workload-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:50.460339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.460322    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d24e2aab-e800-4304-9fab-f66e9f5f11aa" (UID: "d24e2aab-e800-4304-9fab-f66e9f5f11aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:31:50.460585 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.460330    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istiod-ca-cert\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:50.460585 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.460372    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istio-data\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:50.460585 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.460386    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-workload-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:50.462177 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.462151    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istio-token" (OuterVolumeSpecName: "istio-token") pod "d24e2aab-e800-4304-9fab-f66e9f5f11aa" (UID: "d24e2aab-e800-4304-9fab-f66e9f5f11aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:31:50.462306 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.462247    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d24e2aab-e800-4304-9fab-f66e9f5f11aa" (UID: "d24e2aab-e800-4304-9fab-f66e9f5f11aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:31:50.462365 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.462334    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d24e2aab-e800-4304-9fab-f66e9f5f11aa" (UID: "d24e2aab-e800-4304-9fab-f66e9f5f11aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 13:31:50.462365 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.462345    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24e2aab-e800-4304-9fab-f66e9f5f11aa-kube-api-access-stw4v" (OuterVolumeSpecName: "kube-api-access-stw4v") pod "d24e2aab-e800-4304-9fab-f66e9f5f11aa" (UID: "d24e2aab-e800-4304-9fab-f66e9f5f11aa"). InnerVolumeSpecName "kube-api-access-stw4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:31:50.561898 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.561861    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istio-podinfo\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:50.561898 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.561893    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istio-envoy\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:50.561898 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.561904    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stw4v\" (UniqueName: \"kubernetes.io/projected/d24e2aab-e800-4304-9fab-f66e9f5f11aa-kube-api-access-stw4v\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:50.562146 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.561920    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d24e2aab-e800-4304-9fab-f66e9f5f11aa-istio-token\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:50.562146 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:50.561930    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24e2aab-e800-4304-9fab-f66e9f5f11aa-credential-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:51.253132 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:51.253098    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"
Sep 18 13:31:51.253587 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:51.253138    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs" event={"ID":"d24e2aab-e800-4304-9fab-f66e9f5f11aa","Type":"ContainerDied","Data":"948ec587eb502884eb2c90dec85dcfb5e66bd00f7b5168dd77a6ef97e5f9d2c5"}
Sep 18 13:31:51.253587 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:51.253194    2566 scope.go:117] "RemoveContainer" containerID="9845b40193c2ff50f59474ba61c0d6ecb4d6481580ff558a29e6790195f2e45a"
Sep 18 13:31:51.294461 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:51.294426    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"]
Sep 18 13:31:51.312799 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:51.312764    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-cc741da4-openshift-default-7f57475948-48fvs"]
Sep 18 13:31:52.076572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:52.076542    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d24e2aab-e800-4304-9fab-f66e9f5f11aa" path="/var/lib/kubelet/pods/d24e2aab-e800-4304-9fab-f66e9f5f11aa/volumes"
Sep 18 13:31:54.271196 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.271162    2566 generic.go:356] "Generic (PLEG): container finished" podID="1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" containerID="d87c83bdb08e2090753e89efac5e6ad0ce01de2b6b6f3495e3d32191f389bb15" exitCode=0
Sep 18 13:31:54.271666 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.271239    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc" event={"ID":"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20","Type":"ContainerDied","Data":"d87c83bdb08e2090753e89efac5e6ad0ce01de2b6b6f3495e3d32191f389bb15"}
Sep 18 13:31:54.271666 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.271324    2566 scope.go:117] "RemoveContainer" containerID="6093db13ce1ea90b7ea71f2a0b912624c551d7ddfe48d1ba9724c86a46654173"
Sep 18 13:31:54.321449 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.321420    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:31:54.503121 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.503084    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-workload-socket\") pod \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") "
Sep 18 13:31:54.503121 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.503123    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-credential-socket\") pod \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") "
Sep 18 13:31:54.503395 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.503194    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istio-envoy\") pod \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") "
Sep 18 13:31:54.503395 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.503234    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-workload-certs\") pod \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") "
Sep 18 13:31:54.503395 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.503289    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istiod-ca-cert\") pod \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") "
Sep 18 13:31:54.503395 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.503335    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istio-data\") pod \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") "
Sep 18 13:31:54.503395 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.503374    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istio-token\") pod \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") "
Sep 18 13:31:54.503697 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.503404    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkt29\" (UniqueName: \"kubernetes.io/projected/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-kube-api-access-mkt29\") pod \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") "
Sep 18 13:31:54.503697 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.503430    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istio-podinfo\") pod \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\" (UID: \"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20\") "
Sep 18 13:31:54.503697 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.503456    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" (UID: "1dca7ee1-59ae-440b-b6fa-b1dc1d32be20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:31:54.503697 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.503509    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" (UID: "1dca7ee1-59ae-440b-b6fa-b1dc1d32be20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:31:54.503697 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.503639    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" (UID: "1dca7ee1-59ae-440b-b6fa-b1dc1d32be20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:31:54.503955 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.503777    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" (UID: "1dca7ee1-59ae-440b-b6fa-b1dc1d32be20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:31:54.503955 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.503839    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-workload-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:54.503955 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.503859    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-workload-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:54.503955 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.503874    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-credential-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:54.504141 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.503994    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istio-data" (OuterVolumeSpecName: "istio-data") pod "1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" (UID: "1dca7ee1-59ae-440b-b6fa-b1dc1d32be20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:31:54.505552 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.505525    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-kube-api-access-mkt29" (OuterVolumeSpecName: "kube-api-access-mkt29") pod "1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" (UID: "1dca7ee1-59ae-440b-b6fa-b1dc1d32be20"). InnerVolumeSpecName "kube-api-access-mkt29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:31:54.505552 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.505534    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" (UID: "1dca7ee1-59ae-440b-b6fa-b1dc1d32be20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:31:54.506051 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.506035    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istio-token" (OuterVolumeSpecName: "istio-token") pod "1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" (UID: "1dca7ee1-59ae-440b-b6fa-b1dc1d32be20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:31:54.506104 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.506052    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" (UID: "1dca7ee1-59ae-440b-b6fa-b1dc1d32be20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 13:31:54.604755 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.604708    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istiod-ca-cert\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:54.604755 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.604746    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istio-data\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:54.604755 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.604758    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istio-token\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:54.605013 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.604770    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkt29\" (UniqueName: \"kubernetes.io/projected/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-kube-api-access-mkt29\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:54.605013 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.604783    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istio-podinfo\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:54.605013 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:54.604794    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20-istio-envoy\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:31:55.285449 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:55.285418    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"
Sep 18 13:31:55.285449 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:55.285417    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc" event={"ID":"1dca7ee1-59ae-440b-b6fa-b1dc1d32be20","Type":"ContainerDied","Data":"da84f9253bb3257148b6b63cb105e29945aac7428e2301d659d5d0812aea1433"}
Sep 18 13:31:55.285963 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:55.285528    2566 scope.go:117] "RemoveContainer" containerID="d87c83bdb08e2090753e89efac5e6ad0ce01de2b6b6f3495e3d32191f389bb15"
Sep 18 13:31:55.321167 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:55.321122    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"]
Sep 18 13:31:55.326034 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:55.326001    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-b7292e-openshift-default-5b84f9d48-dfbhc"]
Sep 18 13:31:56.076736 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:31:56.076703    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" path="/var/lib/kubelet/pods/1dca7ee1-59ae-440b-b6fa-b1dc1d32be20/volumes"
Sep 18 13:32:03.637220 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.637182    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk"]
Sep 18 13:32:03.637644 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.637575    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" containerName="istio-proxy"
Sep 18 13:32:03.637644 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.637586    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" containerName="istio-proxy"
Sep 18 13:32:03.637644 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.637608    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d24e2aab-e800-4304-9fab-f66e9f5f11aa" containerName="istio-proxy"
Sep 18 13:32:03.637644 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.637614    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d24e2aab-e800-4304-9fab-f66e9f5f11aa" containerName="istio-proxy"
Sep 18 13:32:03.637644 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.637632    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d24e2aab-e800-4304-9fab-f66e9f5f11aa" containerName="istio-proxy"
Sep 18 13:32:03.637644 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.637638    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d24e2aab-e800-4304-9fab-f66e9f5f11aa" containerName="istio-proxy"
Sep 18 13:32:03.637844 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.637654    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55404ea4-fcd5-4371-ba3e-fa3a8c3929fa" containerName="revoke-keys"
Sep 18 13:32:03.637844 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.637659    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="55404ea4-fcd5-4371-ba3e-fa3a8c3929fa" containerName="revoke-keys"
Sep 18 13:32:03.637844 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.637713    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="55404ea4-fcd5-4371-ba3e-fa3a8c3929fa" containerName="revoke-keys"
Sep 18 13:32:03.637844 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.637720    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" containerName="istio-proxy"
Sep 18 13:32:03.637844 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.637726    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d24e2aab-e800-4304-9fab-f66e9f5f11aa" containerName="istio-proxy"
Sep 18 13:32:03.642401 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.642384    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk"
Sep 18 13:32:03.644503 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.644481    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-808e44-dockercfg-gwxwl\""
Sep 18 13:32:03.644795 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.644776    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-808e44\""
Sep 18 13:32:03.661129 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.661101    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk"]
Sep 18 13:32:03.685513 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.685472    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x874w\" (UniqueName: \"kubernetes.io/projected/e4493d69-0aa2-4774-868d-eb5b50750dec-kube-api-access-x874w\") pod \"payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk\" (UID: \"e4493d69-0aa2-4774-868d-eb5b50750dec\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk"
Sep 18 13:32:03.685513 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.685516    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4493d69-0aa2-4774-868d-eb5b50750dec-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk\" (UID: \"e4493d69-0aa2-4774-868d-eb5b50750dec\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk"
Sep 18 13:32:03.704393 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.704363    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-808e44-964c9b4df-bngfr"]
Sep 18 13:32:03.704816 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.704802    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" containerName="istio-proxy"
Sep 18 13:32:03.704892 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.704818    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" containerName="istio-proxy"
Sep 18 13:32:03.704892 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.704890    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d24e2aab-e800-4304-9fab-f66e9f5f11aa" containerName="istio-proxy"
Sep 18 13:32:03.704968 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.704905    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dca7ee1-59ae-440b-b6fa-b1dc1d32be20" containerName="istio-proxy"
Sep 18 13:32:03.708166 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.708139    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-808e44-964c9b4df-bngfr"
Sep 18 13:32:03.720285 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.720198    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-808e44-964c9b4df-bngfr"]
Sep 18 13:32:03.787014 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.786979    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afc58d4-b6e2-4a58-9bc4-ad08e15bef34-plugins-config-volume\") pod \"payload-processing-e2e-explicit-808e44-964c9b4df-bngfr\" (UID: \"5afc58d4-b6e2-4a58-9bc4-ad08e15bef34\") " pod="openshift-ingress/payload-processing-e2e-explicit-808e44-964c9b4df-bngfr"
Sep 18 13:32:03.787231 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.787046    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbzh8\" (UniqueName: \"kubernetes.io/projected/5afc58d4-b6e2-4a58-9bc4-ad08e15bef34-kube-api-access-zbzh8\") pod \"payload-processing-e2e-explicit-808e44-964c9b4df-bngfr\" (UID: \"5afc58d4-b6e2-4a58-9bc4-ad08e15bef34\") " pod="openshift-ingress/payload-processing-e2e-explicit-808e44-964c9b4df-bngfr"
Sep 18 13:32:03.787231 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.787114    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x874w\" (UniqueName: \"kubernetes.io/projected/e4493d69-0aa2-4774-868d-eb5b50750dec-kube-api-access-x874w\") pod \"payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk\" (UID: \"e4493d69-0aa2-4774-868d-eb5b50750dec\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk"
Sep 18 13:32:03.787231 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.787140    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4493d69-0aa2-4774-868d-eb5b50750dec-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk\" (UID: \"e4493d69-0aa2-4774-868d-eb5b50750dec\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk"
Sep 18 13:32:03.787920 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.787895    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4493d69-0aa2-4774-868d-eb5b50750dec-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk\" (UID: \"e4493d69-0aa2-4774-868d-eb5b50750dec\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk"
Sep 18 13:32:03.800588 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.800553    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x874w\" (UniqueName: \"kubernetes.io/projected/e4493d69-0aa2-4774-868d-eb5b50750dec-kube-api-access-x874w\") pod \"payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk\" (UID: \"e4493d69-0aa2-4774-868d-eb5b50750dec\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk"
Sep 18 13:32:03.887793 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.887704    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afc58d4-b6e2-4a58-9bc4-ad08e15bef34-plugins-config-volume\") pod \"payload-processing-e2e-explicit-808e44-964c9b4df-bngfr\" (UID: \"5afc58d4-b6e2-4a58-9bc4-ad08e15bef34\") " pod="openshift-ingress/payload-processing-e2e-explicit-808e44-964c9b4df-bngfr"
Sep 18 13:32:03.887793 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.887764    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbzh8\" (UniqueName: \"kubernetes.io/projected/5afc58d4-b6e2-4a58-9bc4-ad08e15bef34-kube-api-access-zbzh8\") pod \"payload-processing-e2e-explicit-808e44-964c9b4df-bngfr\" (UID: \"5afc58d4-b6e2-4a58-9bc4-ad08e15bef34\") " pod="openshift-ingress/payload-processing-e2e-explicit-808e44-964c9b4df-bngfr"
Sep 18 13:32:03.888551 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.888530    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afc58d4-b6e2-4a58-9bc4-ad08e15bef34-plugins-config-volume\") pod \"payload-processing-e2e-explicit-808e44-964c9b4df-bngfr\" (UID: \"5afc58d4-b6e2-4a58-9bc4-ad08e15bef34\") " pod="openshift-ingress/payload-processing-e2e-explicit-808e44-964c9b4df-bngfr"
Sep 18 13:32:03.900207 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.899751    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbzh8\" (UniqueName: \"kubernetes.io/projected/5afc58d4-b6e2-4a58-9bc4-ad08e15bef34-kube-api-access-zbzh8\") pod \"payload-processing-e2e-explicit-808e44-964c9b4df-bngfr\" (UID: \"5afc58d4-b6e2-4a58-9bc4-ad08e15bef34\") " pod="openshift-ingress/payload-processing-e2e-explicit-808e44-964c9b4df-bngfr"
Sep 18 13:32:03.953313 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:03.953279    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk"
Sep 18 13:32:04.029027 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:04.028993    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-808e44-964c9b4df-bngfr"
Sep 18 13:32:04.234225 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:04.234190    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk"]
Sep 18 13:32:04.235713 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:32:04.235680    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4493d69_0aa2_4774_868d_eb5b50750dec.slice/crio-625c3e05465ed5be4f7ead53c299881fdcfb391078ea72b87c89601fc08650bd WatchSource:0}: Error finding container 625c3e05465ed5be4f7ead53c299881fdcfb391078ea72b87c89601fc08650bd: Status 404 returned error can't find the container with id 625c3e05465ed5be4f7ead53c299881fdcfb391078ea72b87c89601fc08650bd
Sep 18 13:32:04.311095 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:04.311045    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-808e44-964c9b4df-bngfr"]
Sep 18 13:32:04.313342 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:32:04.313314    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5afc58d4_b6e2_4a58_9bc4_ad08e15bef34.slice/crio-f6200af4620f8fb298712c2c5d6c8024e8d0fc5c27044a02c6edefa62596fe8c WatchSource:0}: Error finding container f6200af4620f8fb298712c2c5d6c8024e8d0fc5c27044a02c6edefa62596fe8c: Status 404 returned error can't find the container with id f6200af4620f8fb298712c2c5d6c8024e8d0fc5c27044a02c6edefa62596fe8c
Sep 18 13:32:04.339532 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:04.339497    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk" event={"ID":"e4493d69-0aa2-4774-868d-eb5b50750dec","Type":"ContainerStarted","Data":"b08abfa9f7459759d8f42115805b369a7bd2513ee34af389a580d146a9f2b2ca"}
Sep 18 13:32:04.339687 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:04.339563    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk" event={"ID":"e4493d69-0aa2-4774-868d-eb5b50750dec","Type":"ContainerStarted","Data":"625c3e05465ed5be4f7ead53c299881fdcfb391078ea72b87c89601fc08650bd"}
Sep 18 13:32:04.339753 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:04.339713    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk"
Sep 18 13:32:04.340896 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:04.340870    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-808e44-964c9b4df-bngfr" event={"ID":"5afc58d4-b6e2-4a58-9bc4-ad08e15bef34","Type":"ContainerStarted","Data":"f6200af4620f8fb298712c2c5d6c8024e8d0fc5c27044a02c6edefa62596fe8c"}
Sep 18 13:32:04.363107 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:04.363056    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk" podStartSLOduration=1.363039732 podStartE2EDuration="1.363039732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:32:04.361302986 +0000 UTC m=+1032.796179397" watchObservedRunningTime="2026-09-18 13:32:04.363039732 +0000 UTC m=+1032.797916135"
Sep 18 13:32:05.353523 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:05.353481    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-808e44-964c9b4df-bngfr" event={"ID":"5afc58d4-b6e2-4a58-9bc4-ad08e15bef34","Type":"ContainerStarted","Data":"14776fdd28417ba3b60b8c0594599aed2402d8412344ceb4515efbd41f31a45a"}
Sep 18 13:32:05.353983 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:05.353797    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-808e44-964c9b4df-bngfr"
Sep 18 13:32:05.372348 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:05.372287    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-808e44-964c9b4df-bngfr" podStartSLOduration=2.372266674 podStartE2EDuration="2.372266674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:32:05.370499555 +0000 UTC m=+1033.805375958" watchObservedRunningTime="2026-09-18 13:32:05.372266674 +0000 UTC m=+1033.807143080"
Sep 18 13:32:15.355461 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:15.355429    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk"
Sep 18 13:32:16.362165 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:16.362129    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-808e44-964c9b4df-bngfr"
Sep 18 13:32:46.573362 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.573287    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"]
Sep 18 13:32:46.576953 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.576932    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:32:46.579275 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.579254    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-021eb8\"/\"default-dockercfg-ll268\""
Sep 18 13:32:46.579943 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.579928    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-021eb8\"/\"kube-root-ca.crt\""
Sep 18 13:32:46.580307 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.580289    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-021eb8\"/\"test-model-021eb8-kserve-self-signed-certs\""
Sep 18 13:32:46.580629 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.580589    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-021eb8\"/\"openshift-service-ca.crt\""
Sep 18 13:32:46.592912 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.592888    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"]
Sep 18 13:32:46.701494 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.701463    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pppq\" (UniqueName: \"kubernetes.io/projected/6cab385d-33de-463b-8886-ab159b73701b-kube-api-access-8pppq\") pod \"test-model-021eb8-kserve-84f4f4c465-67gtc\" (UID: \"6cab385d-33de-463b-8886-ab159b73701b\") " pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:32:46.701695 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.701515    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cab385d-33de-463b-8886-ab159b73701b-model-cache\") pod \"test-model-021eb8-kserve-84f4f4c465-67gtc\" (UID: \"6cab385d-33de-463b-8886-ab159b73701b\") " pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:32:46.701695 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.701569    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cab385d-33de-463b-8886-ab159b73701b-tmp-dir\") pod \"test-model-021eb8-kserve-84f4f4c465-67gtc\" (UID: \"6cab385d-33de-463b-8886-ab159b73701b\") " pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:32:46.701695 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.701684    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cab385d-33de-463b-8886-ab159b73701b-dshm\") pod \"test-model-021eb8-kserve-84f4f4c465-67gtc\" (UID: \"6cab385d-33de-463b-8886-ab159b73701b\") " pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:32:46.701860 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.701740    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cab385d-33de-463b-8886-ab159b73701b-tls-certs\") pod \"test-model-021eb8-kserve-84f4f4c465-67gtc\" (UID: \"6cab385d-33de-463b-8886-ab159b73701b\") " pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:32:46.701860 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.701766    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cab385d-33de-463b-8886-ab159b73701b-home\") pod \"test-model-021eb8-kserve-84f4f4c465-67gtc\" (UID: \"6cab385d-33de-463b-8886-ab159b73701b\") " pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:32:46.802916 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.802867    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cab385d-33de-463b-8886-ab159b73701b-model-cache\") pod \"test-model-021eb8-kserve-84f4f4c465-67gtc\" (UID: \"6cab385d-33de-463b-8886-ab159b73701b\") " pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:32:46.802916 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.802920    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cab385d-33de-463b-8886-ab159b73701b-tmp-dir\") pod \"test-model-021eb8-kserve-84f4f4c465-67gtc\" (UID: \"6cab385d-33de-463b-8886-ab159b73701b\") " pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:32:46.803174 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.802965    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cab385d-33de-463b-8886-ab159b73701b-dshm\") pod \"test-model-021eb8-kserve-84f4f4c465-67gtc\" (UID: \"6cab385d-33de-463b-8886-ab159b73701b\") " pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:32:46.803174 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.803007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cab385d-33de-463b-8886-ab159b73701b-tls-certs\") pod \"test-model-021eb8-kserve-84f4f4c465-67gtc\" (UID: \"6cab385d-33de-463b-8886-ab159b73701b\") " pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:32:46.803174 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.803030    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cab385d-33de-463b-8886-ab159b73701b-home\") pod \"test-model-021eb8-kserve-84f4f4c465-67gtc\" (UID: \"6cab385d-33de-463b-8886-ab159b73701b\") " pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:32:46.803174 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.803086    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pppq\" (UniqueName: \"kubernetes.io/projected/6cab385d-33de-463b-8886-ab159b73701b-kube-api-access-8pppq\") pod \"test-model-021eb8-kserve-84f4f4c465-67gtc\" (UID: \"6cab385d-33de-463b-8886-ab159b73701b\") " pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:32:46.803368 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.803342    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cab385d-33de-463b-8886-ab159b73701b-model-cache\") pod \"test-model-021eb8-kserve-84f4f4c465-67gtc\" (UID: \"6cab385d-33de-463b-8886-ab159b73701b\") " pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:32:46.803418 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.803374    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cab385d-33de-463b-8886-ab159b73701b-tmp-dir\") pod \"test-model-021eb8-kserve-84f4f4c465-67gtc\" (UID: \"6cab385d-33de-463b-8886-ab159b73701b\") " pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:32:46.803456 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.803427    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cab385d-33de-463b-8886-ab159b73701b-home\") pod \"test-model-021eb8-kserve-84f4f4c465-67gtc\" (UID: \"6cab385d-33de-463b-8886-ab159b73701b\") " pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:32:46.805292 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.805274    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cab385d-33de-463b-8886-ab159b73701b-dshm\") pod \"test-model-021eb8-kserve-84f4f4c465-67gtc\" (UID: \"6cab385d-33de-463b-8886-ab159b73701b\") " pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:32:46.805664 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.805646    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cab385d-33de-463b-8886-ab159b73701b-tls-certs\") pod \"test-model-021eb8-kserve-84f4f4c465-67gtc\" (UID: \"6cab385d-33de-463b-8886-ab159b73701b\") " pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:32:46.812941 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.812916    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pppq\" (UniqueName: \"kubernetes.io/projected/6cab385d-33de-463b-8886-ab159b73701b-kube-api-access-8pppq\") pod \"test-model-021eb8-kserve-84f4f4c465-67gtc\" (UID: \"6cab385d-33de-463b-8886-ab159b73701b\") " pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:32:46.887297 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:46.887253    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:32:47.182892 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:47.182793    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"]
Sep 18 13:32:47.188293 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:32:47.188263    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cab385d_33de_463b_8886_ab159b73701b.slice/crio-e0e4e148fe45cafca5c4eeba158ae6bc528b8f87d7b2f9bb3dce82a8d2761609 WatchSource:0}: Error finding container e0e4e148fe45cafca5c4eeba158ae6bc528b8f87d7b2f9bb3dce82a8d2761609: Status 404 returned error can't find the container with id e0e4e148fe45cafca5c4eeba158ae6bc528b8f87d7b2f9bb3dce82a8d2761609
Sep 18 13:32:47.592848 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:47.592813    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc" event={"ID":"6cab385d-33de-463b-8886-ab159b73701b","Type":"ContainerStarted","Data":"1ad699a0170b9eeda37c8af4ae3c9021e656c732667ca9ae16336d0e402a6a30"}
Sep 18 13:32:47.593233 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:47.592859    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc" event={"ID":"6cab385d-33de-463b-8886-ab159b73701b","Type":"ContainerStarted","Data":"e0e4e148fe45cafca5c4eeba158ae6bc528b8f87d7b2f9bb3dce82a8d2761609"}
Sep 18 13:32:47.619670 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:47.619611    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc" podStartSLOduration=1.6195780480000002 podStartE2EDuration="1.619578048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:32:47.618139045 +0000 UTC m=+1076.053015462" watchObservedRunningTime="2026-09-18 13:32:47.619578048 +0000 UTC m=+1076.054454454"
Sep 18 13:32:56.888031 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:56.887994    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:32:56.888491 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:56.888151    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:32:56.900696 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:56.900664    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:32:57.663195 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:32:57.663163    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:33:31.679658 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.679623    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"]
Sep 18 13:33:31.683339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.683319    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.686091 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.686069    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-bb3ce27c-openshift-default-dockercfg-5l5v6\""
Sep 18 13:33:31.709135 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.709098    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"]
Sep 18 13:33:31.740450 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.740417    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-workload-certs\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.740695 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.740470    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istio-podinfo\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.740695 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.740496    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-workload-socket\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.740695 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.740566    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istio-token\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.740695 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.740633    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp56n\" (UniqueName: \"kubernetes.io/projected/cf39e7e4-3643-4489-887c-22ce83a4f5f1-kube-api-access-wp56n\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.740695 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.740667    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-credential-socket\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.740903 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.740702    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istio-data\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.740903 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.740833    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istio-envoy\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.740903 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.740865    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istiod-ca-cert\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.841849 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.841815    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-workload-certs\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.842058 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.841857    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istio-podinfo\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.842058 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.841876    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-workload-socket\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.842058 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.841912    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istio-token\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.842058 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.841936    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp56n\" (UniqueName: \"kubernetes.io/projected/cf39e7e4-3643-4489-887c-22ce83a4f5f1-kube-api-access-wp56n\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.842058 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.841961    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-credential-socket\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.842058 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.841999    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istio-data\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.842388 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.842089    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istio-envoy\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.842388 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.842116    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istiod-ca-cert\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.842388 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.842315    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-workload-socket\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.842501 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.842432    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-workload-certs\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.842501 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.842477    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istio-data\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.842790 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.842771    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-credential-socket\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.842862 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.842796    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istiod-ca-cert\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.844430 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.844409    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istio-podinfo\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.844541 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.844433    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istio-envoy\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.852301 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.852266    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp56n\" (UniqueName: \"kubernetes.io/projected/cf39e7e4-3643-4489-887c-22ce83a4f5f1-kube-api-access-wp56n\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.852488 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.852469    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istio-token\") pod \"e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") " pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:31.993827 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:31.993739    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:32.633223 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:32.633197    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"]
Sep 18 13:33:32.635483 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:33:32.635455    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf39e7e4_3643_4489_887c_22ce83a4f5f1.slice/crio-ce7d792d1a2c3ee2a8596fb5a5f82998f1b6ebe6f8a9fa57b2917a3c8d73f102 WatchSource:0}: Error finding container ce7d792d1a2c3ee2a8596fb5a5f82998f1b6ebe6f8a9fa57b2917a3c8d73f102: Status 404 returned error can't find the container with id ce7d792d1a2c3ee2a8596fb5a5f82998f1b6ebe6f8a9fa57b2917a3c8d73f102
Sep 18 13:33:32.637923 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:32.637878    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:33:32.638003 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:32.637951    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:33:32.638003 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:32.637980    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:33:32.848313 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:32.848274    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt" event={"ID":"cf39e7e4-3643-4489-887c-22ce83a4f5f1","Type":"ContainerStarted","Data":"5c10c690e743e773e7875cd4a36c638b539af7972694df02372c1da2ca5ea81f"}
Sep 18 13:33:32.848708 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:32.848342    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt" event={"ID":"cf39e7e4-3643-4489-887c-22ce83a4f5f1","Type":"ContainerStarted","Data":"ce7d792d1a2c3ee2a8596fb5a5f82998f1b6ebe6f8a9fa57b2917a3c8d73f102"}
Sep 18 13:33:32.870237 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:32.870163    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt" podStartSLOduration=1.870141518 podStartE2EDuration="1.870141518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:33:32.868988574 +0000 UTC m=+1121.303864978" watchObservedRunningTime="2026-09-18 13:33:32.870141518 +0000 UTC m=+1121.305017924"
Sep 18 13:33:32.994561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:32.994456    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:33.999997 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:33.999964    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:34.858873 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:34.858846    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:34.860054 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:34.860032    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:33:45.387407 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:45.387371    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv"]
Sep 18 13:33:45.391154 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:45.391134    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv"
Sep 18 13:33:45.393874 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:45.393852    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-bb3ce27c-dockercfg-mfsq4\""
Sep 18 13:33:45.394320 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:45.394305    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-bb3ce27c\""
Sep 18 13:33:45.407654 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:45.407626    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv"]
Sep 18 13:33:45.482475 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:45.479916    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee82569e-6947-4a9e-b449-be32dde6bcaa-plugins-config-volume\") pod \"payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv\" (UID: \"ee82569e-6947-4a9e-b449-be32dde6bcaa\") " pod="openshift-ingress/payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv"
Sep 18 13:33:45.482475 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:45.480096    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzwqh\" (UniqueName: \"kubernetes.io/projected/ee82569e-6947-4a9e-b449-be32dde6bcaa-kube-api-access-lzwqh\") pod \"payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv\" (UID: \"ee82569e-6947-4a9e-b449-be32dde6bcaa\") " pod="openshift-ingress/payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv"
Sep 18 13:33:45.581002 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:45.580961    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee82569e-6947-4a9e-b449-be32dde6bcaa-plugins-config-volume\") pod \"payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv\" (UID: \"ee82569e-6947-4a9e-b449-be32dde6bcaa\") " pod="openshift-ingress/payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv"
Sep 18 13:33:45.581182 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:45.581026    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzwqh\" (UniqueName: \"kubernetes.io/projected/ee82569e-6947-4a9e-b449-be32dde6bcaa-kube-api-access-lzwqh\") pod \"payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv\" (UID: \"ee82569e-6947-4a9e-b449-be32dde6bcaa\") " pod="openshift-ingress/payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv"
Sep 18 13:33:45.581686 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:45.581667    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee82569e-6947-4a9e-b449-be32dde6bcaa-plugins-config-volume\") pod \"payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv\" (UID: \"ee82569e-6947-4a9e-b449-be32dde6bcaa\") " pod="openshift-ingress/payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv"
Sep 18 13:33:45.589448 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:45.589415    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzwqh\" (UniqueName: \"kubernetes.io/projected/ee82569e-6947-4a9e-b449-be32dde6bcaa-kube-api-access-lzwqh\") pod \"payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv\" (UID: \"ee82569e-6947-4a9e-b449-be32dde6bcaa\") " pod="openshift-ingress/payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv"
Sep 18 13:33:45.701791 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:45.701705    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv"
Sep 18 13:33:45.995897 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:45.995815    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv"]
Sep 18 13:33:45.999360 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:33:45.999330    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee82569e_6947_4a9e_b449_be32dde6bcaa.slice/crio-b7cde00538b7d2477d19915a826e5e68a3099d2618b156c8d64d5137cb2d5e61 WatchSource:0}: Error finding container b7cde00538b7d2477d19915a826e5e68a3099d2618b156c8d64d5137cb2d5e61: Status 404 returned error can't find the container with id b7cde00538b7d2477d19915a826e5e68a3099d2618b156c8d64d5137cb2d5e61
Sep 18 13:33:46.928752 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:46.928712    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv" event={"ID":"ee82569e-6947-4a9e-b449-be32dde6bcaa","Type":"ContainerStarted","Data":"b7cc931973ff860105242ab1bbde4c0903b6e4d8b37c2139dbf99f3a6d072ca7"}
Sep 18 13:33:46.929239 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:46.928772    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv" event={"ID":"ee82569e-6947-4a9e-b449-be32dde6bcaa","Type":"ContainerStarted","Data":"b7cde00538b7d2477d19915a826e5e68a3099d2618b156c8d64d5137cb2d5e61"}
Sep 18 13:33:46.929239 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:46.928802    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv"
Sep 18 13:33:46.949207 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:46.949144    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv" podStartSLOduration=1.9491233019999998 podStartE2EDuration="1.949123302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:33:46.948478775 +0000 UTC m=+1135.383355179" watchObservedRunningTime="2026-09-18 13:33:46.949123302 +0000 UTC m=+1135.383999720"
Sep 18 13:33:57.936573 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:33:57.936539    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv"
Sep 18 13:34:03.073745 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:03.073706    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-808e44-964c9b4df-bngfr"]
Sep 18 13:34:03.074236 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:03.073922    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-808e44-964c9b4df-bngfr" podUID="5afc58d4-b6e2-4a58-9bc4-ad08e15bef34" containerName="payload-processing" containerID="cri-o://14776fdd28417ba3b60b8c0594599aed2402d8412344ceb4515efbd41f31a45a" gracePeriod=30
Sep 18 13:34:03.089122 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:03.089088    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk"]
Sep 18 13:34:03.089705 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:03.089329    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk" podUID="e4493d69-0aa2-4774-868d-eb5b50750dec" containerName="payload-pre-processing" containerID="cri-o://b08abfa9f7459759d8f42115805b369a7bd2513ee34af389a580d146a9f2b2ca" gracePeriod=30
Sep 18 13:34:03.363889 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:03.363860    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-808e44-964c9b4df-bngfr"
Sep 18 13:34:03.399246 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:03.399207    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk"
Sep 18 13:34:03.465532 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:03.465489    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x874w\" (UniqueName: \"kubernetes.io/projected/e4493d69-0aa2-4774-868d-eb5b50750dec-kube-api-access-x874w\") pod \"e4493d69-0aa2-4774-868d-eb5b50750dec\" (UID: \"e4493d69-0aa2-4774-868d-eb5b50750dec\") "
Sep 18 13:34:03.465764 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:03.465541    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbzh8\" (UniqueName: \"kubernetes.io/projected/5afc58d4-b6e2-4a58-9bc4-ad08e15bef34-kube-api-access-zbzh8\") pod \"5afc58d4-b6e2-4a58-9bc4-ad08e15bef34\" (UID: \"5afc58d4-b6e2-4a58-9bc4-ad08e15bef34\") "
Sep 18 13:34:03.465764 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:03.465581    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4493d69-0aa2-4774-868d-eb5b50750dec-plugins-config-volume\") pod \"e4493d69-0aa2-4774-868d-eb5b50750dec\" (UID: \"e4493d69-0aa2-4774-868d-eb5b50750dec\") "
Sep 18 13:34:03.465764 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:03.465625    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afc58d4-b6e2-4a58-9bc4-ad08e15bef34-plugins-config-volume\") pod \"5afc58d4-b6e2-4a58-9bc4-ad08e15bef34\" (UID: \"5afc58d4-b6e2-4a58-9bc4-ad08e15bef34\") "
Sep 18 13:34:03.466013 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:03.465986    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4493d69-0aa2-4774-868d-eb5b50750dec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4493d69-0aa2-4774-868d-eb5b50750dec" (UID: "e4493d69-0aa2-4774-868d-eb5b50750dec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:34:03.466354 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:03.466324    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5afc58d4-b6e2-4a58-9bc4-ad08e15bef34-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5afc58d4-b6e2-4a58-9bc4-ad08e15bef34" (UID: "5afc58d4-b6e2-4a58-9bc4-ad08e15bef34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:34:03.467845 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:03.467822    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4493d69-0aa2-4774-868d-eb5b50750dec-kube-api-access-x874w" (OuterVolumeSpecName: "kube-api-access-x874w") pod "e4493d69-0aa2-4774-868d-eb5b50750dec" (UID: "e4493d69-0aa2-4774-868d-eb5b50750dec"). InnerVolumeSpecName "kube-api-access-x874w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:34:03.468362 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:03.468349    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5afc58d4-b6e2-4a58-9bc4-ad08e15bef34-kube-api-access-zbzh8" (OuterVolumeSpecName: "kube-api-access-zbzh8") pod "5afc58d4-b6e2-4a58-9bc4-ad08e15bef34" (UID: "5afc58d4-b6e2-4a58-9bc4-ad08e15bef34"). InnerVolumeSpecName "kube-api-access-zbzh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:34:03.567920 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:03.567879    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4493d69-0aa2-4774-868d-eb5b50750dec-plugins-config-volume\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:34:03.567920 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:03.567917    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afc58d4-b6e2-4a58-9bc4-ad08e15bef34-plugins-config-volume\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:34:03.568120 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:03.567931    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x874w\" (UniqueName: \"kubernetes.io/projected/e4493d69-0aa2-4774-868d-eb5b50750dec-kube-api-access-x874w\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:34:03.568120 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:03.567944    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbzh8\" (UniqueName: \"kubernetes.io/projected/5afc58d4-b6e2-4a58-9bc4-ad08e15bef34-kube-api-access-zbzh8\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:34:04.029560 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:04.029523    2566 generic.go:356] "Generic (PLEG): container finished" podID="e4493d69-0aa2-4774-868d-eb5b50750dec" containerID="b08abfa9f7459759d8f42115805b369a7bd2513ee34af389a580d146a9f2b2ca" exitCode=0
Sep 18 13:34:04.029778 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:04.029568    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk"
Sep 18 13:34:04.029778 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:04.029631    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk" event={"ID":"e4493d69-0aa2-4774-868d-eb5b50750dec","Type":"ContainerDied","Data":"b08abfa9f7459759d8f42115805b369a7bd2513ee34af389a580d146a9f2b2ca"}
Sep 18 13:34:04.029778 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:04.029686    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk" event={"ID":"e4493d69-0aa2-4774-868d-eb5b50750dec","Type":"ContainerDied","Data":"625c3e05465ed5be4f7ead53c299881fdcfb391078ea72b87c89601fc08650bd"}
Sep 18 13:34:04.029778 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:04.029712    2566 scope.go:117] "RemoveContainer" containerID="b08abfa9f7459759d8f42115805b369a7bd2513ee34af389a580d146a9f2b2ca"
Sep 18 13:34:04.031233 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:04.031206    2566 generic.go:356] "Generic (PLEG): container finished" podID="5afc58d4-b6e2-4a58-9bc4-ad08e15bef34" containerID="14776fdd28417ba3b60b8c0594599aed2402d8412344ceb4515efbd41f31a45a" exitCode=0
Sep 18 13:34:04.031356 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:04.031249    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-808e44-964c9b4df-bngfr"
Sep 18 13:34:04.031356 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:04.031247    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-808e44-964c9b4df-bngfr" event={"ID":"5afc58d4-b6e2-4a58-9bc4-ad08e15bef34","Type":"ContainerDied","Data":"14776fdd28417ba3b60b8c0594599aed2402d8412344ceb4515efbd41f31a45a"}
Sep 18 13:34:04.031473 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:04.031367    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-808e44-964c9b4df-bngfr" event={"ID":"5afc58d4-b6e2-4a58-9bc4-ad08e15bef34","Type":"ContainerDied","Data":"f6200af4620f8fb298712c2c5d6c8024e8d0fc5c27044a02c6edefa62596fe8c"}
Sep 18 13:34:04.040448 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:04.040429    2566 scope.go:117] "RemoveContainer" containerID="b08abfa9f7459759d8f42115805b369a7bd2513ee34af389a580d146a9f2b2ca"
Sep 18 13:34:04.040845 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:34:04.040825    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b08abfa9f7459759d8f42115805b369a7bd2513ee34af389a580d146a9f2b2ca\": container with ID starting with b08abfa9f7459759d8f42115805b369a7bd2513ee34af389a580d146a9f2b2ca not found: ID does not exist" containerID="b08abfa9f7459759d8f42115805b369a7bd2513ee34af389a580d146a9f2b2ca"
Sep 18 13:34:04.040922 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:04.040855    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b08abfa9f7459759d8f42115805b369a7bd2513ee34af389a580d146a9f2b2ca"} err="failed to get container status \"b08abfa9f7459759d8f42115805b369a7bd2513ee34af389a580d146a9f2b2ca\": rpc error: code = NotFound desc = could not find container \"b08abfa9f7459759d8f42115805b369a7bd2513ee34af389a580d146a9f2b2ca\": container with ID starting with b08abfa9f7459759d8f42115805b369a7bd2513ee34af389a580d146a9f2b2ca not found: ID does not exist"
Sep 18 13:34:04.040922 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:04.040873    2566 scope.go:117] "RemoveContainer" containerID="14776fdd28417ba3b60b8c0594599aed2402d8412344ceb4515efbd41f31a45a"
Sep 18 13:34:04.050230 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:04.050211    2566 scope.go:117] "RemoveContainer" containerID="14776fdd28417ba3b60b8c0594599aed2402d8412344ceb4515efbd41f31a45a"
Sep 18 13:34:04.050533 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:34:04.050513    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14776fdd28417ba3b60b8c0594599aed2402d8412344ceb4515efbd41f31a45a\": container with ID starting with 14776fdd28417ba3b60b8c0594599aed2402d8412344ceb4515efbd41f31a45a not found: ID does not exist" containerID="14776fdd28417ba3b60b8c0594599aed2402d8412344ceb4515efbd41f31a45a"
Sep 18 13:34:04.050618 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:04.050540    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14776fdd28417ba3b60b8c0594599aed2402d8412344ceb4515efbd41f31a45a"} err="failed to get container status \"14776fdd28417ba3b60b8c0594599aed2402d8412344ceb4515efbd41f31a45a\": rpc error: code = NotFound desc = could not find container \"14776fdd28417ba3b60b8c0594599aed2402d8412344ceb4515efbd41f31a45a\": container with ID starting with 14776fdd28417ba3b60b8c0594599aed2402d8412344ceb4515efbd41f31a45a not found: ID does not exist"
Sep 18 13:34:04.061299 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:04.061271    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-808e44-964c9b4df-bngfr"]
Sep 18 13:34:04.066422 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:04.066398    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-808e44-964c9b4df-bngfr"]
Sep 18 13:34:04.077971 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:04.077943    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5afc58d4-b6e2-4a58-9bc4-ad08e15bef34" path="/var/lib/kubelet/pods/5afc58d4-b6e2-4a58-9bc4-ad08e15bef34/volumes"
Sep 18 13:34:04.083256 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:04.083235    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk"]
Sep 18 13:34:04.087776 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:04.087753    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-808e44-74f94c5686-hk4hk"]
Sep 18 13:34:05.317895 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:05.317865    2566 patch_prober.go:28] interesting pod/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": read tcp 10.133.0.2:39748->10.133.0.56:15021: read: connection reset by peer" start-of-body=
Sep 18 13:34:05.318284 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:05.317989    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt" podUID="cf39e7e4-3643-4489-887c-22ce83a4f5f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": read tcp 10.133.0.2:39748->10.133.0.56:15021: read: connection reset by peer"
Sep 18 13:34:06.046489 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:06.046456    2566 generic.go:356] "Generic (PLEG): container finished" podID="cf39e7e4-3643-4489-887c-22ce83a4f5f1" containerID="5c10c690e743e773e7875cd4a36c638b539af7972694df02372c1da2ca5ea81f" exitCode=0
Sep 18 13:34:06.046685 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:06.046515    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt" event={"ID":"cf39e7e4-3643-4489-887c-22ce83a4f5f1","Type":"ContainerDied","Data":"5c10c690e743e773e7875cd4a36c638b539af7972694df02372c1da2ca5ea81f"}
Sep 18 13:34:06.046935 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:06.046919    2566 scope.go:117] "RemoveContainer" containerID="5c10c690e743e773e7875cd4a36c638b539af7972694df02372c1da2ca5ea81f"
Sep 18 13:34:06.048069 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:06.048043    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:34:06.048130 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:06.048119    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:34:06.048194 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:06.048183    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:34:06.083436 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:06.083400    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4493d69-0aa2-4774-868d-eb5b50750dec" path="/var/lib/kubelet/pods/e4493d69-0aa2-4774-868d-eb5b50750dec/volumes"
Sep 18 13:34:07.059065 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:07.059016    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt" event={"ID":"cf39e7e4-3643-4489-887c-22ce83a4f5f1","Type":"ContainerStarted","Data":"9561441fd410c1aa96c795f5d10324dd9144d5c7b7d1eeaec76377736d1577c5"}
Sep 18 13:34:07.994948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:07.994914    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:34:07.995124 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:07.995094    2566 patch_prober.go:28] interesting pod/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 18 13:34:07.995169 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:07.995140    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt" podUID="cf39e7e4-3643-4489-887c-22ce83a4f5f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 18 13:34:08.116024 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:08.115984    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3ce27c-9dqwf"]
Sep 18 13:34:08.116466 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:08.116424    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4493d69-0aa2-4774-868d-eb5b50750dec" containerName="payload-pre-processing"
Sep 18 13:34:08.116466 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:08.116437    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4493d69-0aa2-4774-868d-eb5b50750dec" containerName="payload-pre-processing"
Sep 18 13:34:08.116466 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:08.116460    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5afc58d4-b6e2-4a58-9bc4-ad08e15bef34" containerName="payload-processing"
Sep 18 13:34:08.116466 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:08.116466    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5afc58d4-b6e2-4a58-9bc4-ad08e15bef34" containerName="payload-processing"
Sep 18 13:34:08.116619 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:08.116535    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4493d69-0aa2-4774-868d-eb5b50750dec" containerName="payload-pre-processing"
Sep 18 13:34:08.116619 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:08.116542    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5afc58d4-b6e2-4a58-9bc4-ad08e15bef34" containerName="payload-processing"
Sep 18 13:34:08.118773 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:08.118757    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3ce27c-9dqwf"
Sep 18 13:34:08.121067 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:08.121045    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nh2cd\""
Sep 18 13:34:08.121545 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:08.121529    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:34:08.131923 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:08.131894    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3ce27c-9dqwf"]
Sep 18 13:34:08.216942 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:08.216906    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a119996-341c-4e76-bb03-7f79c3757379-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bb3ce27c-9dqwf\" (UID: \"5a119996-341c-4e76-bb03-7f79c3757379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3ce27c-9dqwf"
Sep 18 13:34:08.318609 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:08.318508    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a119996-341c-4e76-bb03-7f79c3757379-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bb3ce27c-9dqwf\" (UID: \"5a119996-341c-4e76-bb03-7f79c3757379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3ce27c-9dqwf"
Sep 18 13:34:08.319202 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:08.319181    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a119996-341c-4e76-bb03-7f79c3757379-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-bb3ce27c-9dqwf\" (UID: \"5a119996-341c-4e76-bb03-7f79c3757379\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3ce27c-9dqwf"
Sep 18 13:34:08.429660 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:08.429626    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3ce27c-9dqwf"
Sep 18 13:34:08.734161 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:08.734131    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3ce27c-9dqwf"]
Sep 18 13:34:08.735334 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:34:08.735302    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a119996_341c_4e76_bb03_7f79c3757379.slice/crio-82678fd13f3dceed895cf0c6c33f1c7dc62e019b108559faa933dadfccf842fd WatchSource:0}: Error finding container 82678fd13f3dceed895cf0c6c33f1c7dc62e019b108559faa933dadfccf842fd: Status 404 returned error can't find the container with id 82678fd13f3dceed895cf0c6c33f1c7dc62e019b108559faa933dadfccf842fd
Sep 18 13:34:08.994528 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:08.994443    2566 patch_prober.go:28] interesting pod/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 18 13:34:08.994528 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:08.994497    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt" podUID="cf39e7e4-3643-4489-887c-22ce83a4f5f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 18 13:34:09.075222 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:09.075187    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3ce27c-9dqwf" event={"ID":"5a119996-341c-4e76-bb03-7f79c3757379","Type":"ContainerStarted","Data":"82678fd13f3dceed895cf0c6c33f1c7dc62e019b108559faa933dadfccf842fd"}
Sep 18 13:34:09.994523 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:09.994476    2566 patch_prober.go:28] interesting pod/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 18 13:34:09.995059 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:09.994545    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt" podUID="cf39e7e4-3643-4489-887c-22ce83a4f5f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 18 13:34:10.082386 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:10.082301    2566 generic.go:356] "Generic (PLEG): container finished" podID="5a119996-341c-4e76-bb03-7f79c3757379" containerID="12211cda556a51ed03eb5b8d900fc67b0e8018b9e34cbefd128140b4a78bc2a8" exitCode=0
Sep 18 13:34:10.082540 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:10.082386    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3ce27c-9dqwf" event={"ID":"5a119996-341c-4e76-bb03-7f79c3757379","Type":"ContainerDied","Data":"12211cda556a51ed03eb5b8d900fc67b0e8018b9e34cbefd128140b4a78bc2a8"}
Sep 18 13:34:10.994340 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:10.994302    2566 patch_prober.go:28] interesting pod/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 18 13:34:10.994530 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:10.994354    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt" podUID="cf39e7e4-3643-4489-887c-22ce83a4f5f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 18 13:34:11.269097 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:11.269025    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3ce27c-9dqwf"
Sep 18 13:34:11.370817 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:11.370784    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a119996-341c-4e76-bb03-7f79c3757379-maas-api-service-ca\") pod \"5a119996-341c-4e76-bb03-7f79c3757379\" (UID: \"5a119996-341c-4e76-bb03-7f79c3757379\") "
Sep 18 13:34:11.371137 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:11.371113    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a119996-341c-4e76-bb03-7f79c3757379-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5a119996-341c-4e76-bb03-7f79c3757379" (UID: "5a119996-341c-4e76-bb03-7f79c3757379"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:34:11.472260 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:11.472224    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a119996-341c-4e76-bb03-7f79c3757379-maas-api-service-ca\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:34:11.995043 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:11.995005    2566 patch_prober.go:28] interesting pod/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 18 13:34:11.995245 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:11.995071    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt" podUID="cf39e7e4-3643-4489-887c-22ce83a4f5f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 18 13:34:12.095134 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:12.095104    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3ce27c-9dqwf"
Sep 18 13:34:12.095293 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:12.095148    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3ce27c-9dqwf" event={"ID":"5a119996-341c-4e76-bb03-7f79c3757379","Type":"ContainerDied","Data":"82678fd13f3dceed895cf0c6c33f1c7dc62e019b108559faa933dadfccf842fd"}
Sep 18 13:34:12.095293 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:12.095196    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82678fd13f3dceed895cf0c6c33f1c7dc62e019b108559faa933dadfccf842fd"
Sep 18 13:34:12.997811 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:12.997776    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:34:12.998287 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:12.998172    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:34:12.998845 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:12.998826    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:34:22.339306 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:22.339271    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv"]
Sep 18 13:34:22.339869 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:22.339480    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv" podUID="ee82569e-6947-4a9e-b449-be32dde6bcaa" containerName="payload-processing" containerID="cri-o://b7cc931973ff860105242ab1bbde4c0903b6e4d8b37c2139dbf99f3a6d072ca7" gracePeriod=30
Sep 18 13:34:22.623024 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:22.622996    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv"
Sep 18 13:34:22.623176 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:22.622993    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3ce27c-9dqwf"]
Sep 18 13:34:22.625242 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:22.625213    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-bb3ce27c-9dqwf"]
Sep 18 13:34:22.793447 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:22.793398    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee82569e-6947-4a9e-b449-be32dde6bcaa-plugins-config-volume\") pod \"ee82569e-6947-4a9e-b449-be32dde6bcaa\" (UID: \"ee82569e-6947-4a9e-b449-be32dde6bcaa\") "
Sep 18 13:34:22.793447 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:22.793460    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzwqh\" (UniqueName: \"kubernetes.io/projected/ee82569e-6947-4a9e-b449-be32dde6bcaa-kube-api-access-lzwqh\") pod \"ee82569e-6947-4a9e-b449-be32dde6bcaa\" (UID: \"ee82569e-6947-4a9e-b449-be32dde6bcaa\") "
Sep 18 13:34:22.793939 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:22.793903    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee82569e-6947-4a9e-b449-be32dde6bcaa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ee82569e-6947-4a9e-b449-be32dde6bcaa" (UID: "ee82569e-6947-4a9e-b449-be32dde6bcaa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:34:22.795673 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:22.795650    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee82569e-6947-4a9e-b449-be32dde6bcaa-kube-api-access-lzwqh" (OuterVolumeSpecName: "kube-api-access-lzwqh") pod "ee82569e-6947-4a9e-b449-be32dde6bcaa" (UID: "ee82569e-6947-4a9e-b449-be32dde6bcaa"). InnerVolumeSpecName "kube-api-access-lzwqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:34:22.894816 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:22.894719    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee82569e-6947-4a9e-b449-be32dde6bcaa-plugins-config-volume\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:34:22.894816 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:22.894758    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzwqh\" (UniqueName: \"kubernetes.io/projected/ee82569e-6947-4a9e-b449-be32dde6bcaa-kube-api-access-lzwqh\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:34:23.159402 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:23.159310    2566 generic.go:356] "Generic (PLEG): container finished" podID="ee82569e-6947-4a9e-b449-be32dde6bcaa" containerID="b7cc931973ff860105242ab1bbde4c0903b6e4d8b37c2139dbf99f3a6d072ca7" exitCode=0
Sep 18 13:34:23.159402 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:23.159355    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv"
Sep 18 13:34:23.159586 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:23.159400    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv" event={"ID":"ee82569e-6947-4a9e-b449-be32dde6bcaa","Type":"ContainerDied","Data":"b7cc931973ff860105242ab1bbde4c0903b6e4d8b37c2139dbf99f3a6d072ca7"}
Sep 18 13:34:23.159586 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:23.159452    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv" event={"ID":"ee82569e-6947-4a9e-b449-be32dde6bcaa","Type":"ContainerDied","Data":"b7cde00538b7d2477d19915a826e5e68a3099d2618b156c8d64d5137cb2d5e61"}
Sep 18 13:34:23.159586 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:23.159476    2566 scope.go:117] "RemoveContainer" containerID="b7cc931973ff860105242ab1bbde4c0903b6e4d8b37c2139dbf99f3a6d072ca7"
Sep 18 13:34:23.169373 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:23.169351    2566 scope.go:117] "RemoveContainer" containerID="b7cc931973ff860105242ab1bbde4c0903b6e4d8b37c2139dbf99f3a6d072ca7"
Sep 18 13:34:23.169697 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:34:23.169677    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7cc931973ff860105242ab1bbde4c0903b6e4d8b37c2139dbf99f3a6d072ca7\": container with ID starting with b7cc931973ff860105242ab1bbde4c0903b6e4d8b37c2139dbf99f3a6d072ca7 not found: ID does not exist" containerID="b7cc931973ff860105242ab1bbde4c0903b6e4d8b37c2139dbf99f3a6d072ca7"
Sep 18 13:34:23.169776 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:23.169704    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7cc931973ff860105242ab1bbde4c0903b6e4d8b37c2139dbf99f3a6d072ca7"} err="failed to get container status \"b7cc931973ff860105242ab1bbde4c0903b6e4d8b37c2139dbf99f3a6d072ca7\": rpc error: code = NotFound desc = could not find container \"b7cc931973ff860105242ab1bbde4c0903b6e4d8b37c2139dbf99f3a6d072ca7\": container with ID starting with b7cc931973ff860105242ab1bbde4c0903b6e4d8b37c2139dbf99f3a6d072ca7 not found: ID does not exist"
Sep 18 13:34:23.185489 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:23.185453    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv"]
Sep 18 13:34:23.195379 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:23.195347    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-bb3ce27c-6899968df-kzscv"]
Sep 18 13:34:24.076857 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:24.076827    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a119996-341c-4e76-bb03-7f79c3757379" path="/var/lib/kubelet/pods/5a119996-341c-4e76-bb03-7f79c3757379/volumes"
Sep 18 13:34:24.077265 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:24.077213    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee82569e-6947-4a9e-b449-be32dde6bcaa" path="/var/lib/kubelet/pods/ee82569e-6947-4a9e-b449-be32dde6bcaa/volumes"
Sep 18 13:34:31.352911 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:31.352872    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"]
Sep 18 13:34:31.353326 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:31.353075    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt" podUID="cf39e7e4-3643-4489-887c-22ce83a4f5f1" containerName="istio-proxy" containerID="cri-o://9561441fd410c1aa96c795f5d10324dd9144d5c7b7d1eeaec76377736d1577c5" gracePeriod=30
Sep 18 13:34:36.596403 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.596376    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:34:36.627477 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.627401    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istio-token\") pod \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") "
Sep 18 13:34:36.627477 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.627467    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istiod-ca-cert\") pod \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") "
Sep 18 13:34:36.627756 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.627484    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-workload-certs\") pod \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") "
Sep 18 13:34:36.627756 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.627514    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istio-data\") pod \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") "
Sep 18 13:34:36.627756 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.627565    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-credential-socket\") pod \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") "
Sep 18 13:34:36.627756 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.627738    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wp56n\" (UniqueName: \"kubernetes.io/projected/cf39e7e4-3643-4489-887c-22ce83a4f5f1-kube-api-access-wp56n\") pod \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") "
Sep 18 13:34:36.627961 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.627820    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cf39e7e4-3643-4489-887c-22ce83a4f5f1" (UID: "cf39e7e4-3643-4489-887c-22ce83a4f5f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:34:36.627961 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.627844    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cf39e7e4-3643-4489-887c-22ce83a4f5f1" (UID: "cf39e7e4-3643-4489-887c-22ce83a4f5f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:34:36.627961 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.627857    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istio-data" (OuterVolumeSpecName: "istio-data") pod "cf39e7e4-3643-4489-887c-22ce83a4f5f1" (UID: "cf39e7e4-3643-4489-887c-22ce83a4f5f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:34:36.627961 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.627947    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istio-podinfo\") pod \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") "
Sep 18 13:34:36.628098 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.627962    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cf39e7e4-3643-4489-887c-22ce83a4f5f1" (UID: "cf39e7e4-3643-4489-887c-22ce83a4f5f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:34:36.628098 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.628031    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-workload-socket\") pod \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") "
Sep 18 13:34:36.628098 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.628066    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istio-envoy\") pod \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\" (UID: \"cf39e7e4-3643-4489-887c-22ce83a4f5f1\") "
Sep 18 13:34:36.628326 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.628299    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cf39e7e4-3643-4489-887c-22ce83a4f5f1" (UID: "cf39e7e4-3643-4489-887c-22ce83a4f5f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:34:36.628459 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.628429    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-workload-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:34:36.628459 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.628449    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istiod-ca-cert\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:34:36.628560 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.628463    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-workload-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:34:36.628560 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.628477    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istio-data\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:34:36.628560 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.628490    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-credential-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:34:36.630214 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.630195    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cf39e7e4-3643-4489-887c-22ce83a4f5f1" (UID: "cf39e7e4-3643-4489-887c-22ce83a4f5f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:34:36.630369 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.630340    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf39e7e4-3643-4489-887c-22ce83a4f5f1-kube-api-access-wp56n" (OuterVolumeSpecName: "kube-api-access-wp56n") pod "cf39e7e4-3643-4489-887c-22ce83a4f5f1" (UID: "cf39e7e4-3643-4489-887c-22ce83a4f5f1"). InnerVolumeSpecName "kube-api-access-wp56n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:34:36.630468 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.630363    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istio-token" (OuterVolumeSpecName: "istio-token") pod "cf39e7e4-3643-4489-887c-22ce83a4f5f1" (UID: "cf39e7e4-3643-4489-887c-22ce83a4f5f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:34:36.630468 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.630414    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cf39e7e4-3643-4489-887c-22ce83a4f5f1" (UID: "cf39e7e4-3643-4489-887c-22ce83a4f5f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 13:34:36.729281 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.729231    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp56n\" (UniqueName: \"kubernetes.io/projected/cf39e7e4-3643-4489-887c-22ce83a4f5f1-kube-api-access-wp56n\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:34:36.729281 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.729277    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istio-podinfo\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:34:36.729281 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.729287    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istio-envoy\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:34:36.729281 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:36.729296    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf39e7e4-3643-4489-887c-22ce83a4f5f1-istio-token\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:34:37.111716 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.111684    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"]
Sep 18 13:34:37.112072 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.112060    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf39e7e4-3643-4489-887c-22ce83a4f5f1" containerName="istio-proxy"
Sep 18 13:34:37.112118 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.112074    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf39e7e4-3643-4489-887c-22ce83a4f5f1" containerName="istio-proxy"
Sep 18 13:34:37.112118 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.112089    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee82569e-6947-4a9e-b449-be32dde6bcaa" containerName="payload-processing"
Sep 18 13:34:37.112118 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.112094    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee82569e-6947-4a9e-b449-be32dde6bcaa" containerName="payload-processing"
Sep 18 13:34:37.112118 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.112101    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf39e7e4-3643-4489-887c-22ce83a4f5f1" containerName="istio-proxy"
Sep 18 13:34:37.112118 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.112107    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf39e7e4-3643-4489-887c-22ce83a4f5f1" containerName="istio-proxy"
Sep 18 13:34:37.112118 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.112115    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a119996-341c-4e76-bb03-7f79c3757379" containerName="revoke-keys"
Sep 18 13:34:37.112118 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.112120    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a119996-341c-4e76-bb03-7f79c3757379" containerName="revoke-keys"
Sep 18 13:34:37.112325 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.112187    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf39e7e4-3643-4489-887c-22ce83a4f5f1" containerName="istio-proxy"
Sep 18 13:34:37.112325 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.112195    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee82569e-6947-4a9e-b449-be32dde6bcaa" containerName="payload-processing"
Sep 18 13:34:37.112325 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.112202    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a119996-341c-4e76-bb03-7f79c3757379" containerName="revoke-keys"
Sep 18 13:34:37.114549 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.114530    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:34:37.117625 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.116929    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-5be8a3\"/\"kube-root-ca.crt\""
Sep 18 13:34:37.118309 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.118011    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5be8a3\"/\"default-dockercfg-b7jf8\""
Sep 18 13:34:37.118309 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.118226    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5be8a3\"/\"e2e-conflict-a-158c7039-kserve-self-signed-certs\""
Sep 18 13:34:37.118626 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.118510    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-5be8a3\"/\"openshift-service-ca.crt\""
Sep 18 13:34:37.131868 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.131847    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"]
Sep 18 13:34:37.234156 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.234116    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69b4b46b-eee4-49fc-8165-b9b6d664535f-tls-certs\") pod \"e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5\" (UID: \"69b4b46b-eee4-49fc-8165-b9b6d664535f\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:34:37.235184 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.234223    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxxsn\" (UniqueName: \"kubernetes.io/projected/69b4b46b-eee4-49fc-8165-b9b6d664535f-kube-api-access-rxxsn\") pod \"e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5\" (UID: \"69b4b46b-eee4-49fc-8165-b9b6d664535f\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:34:37.235184 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.234292    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69b4b46b-eee4-49fc-8165-b9b6d664535f-home\") pod \"e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5\" (UID: \"69b4b46b-eee4-49fc-8165-b9b6d664535f\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:34:37.235184 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.234333    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69b4b46b-eee4-49fc-8165-b9b6d664535f-dshm\") pod \"e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5\" (UID: \"69b4b46b-eee4-49fc-8165-b9b6d664535f\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:34:37.235184 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.234398    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69b4b46b-eee4-49fc-8165-b9b6d664535f-model-cache\") pod \"e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5\" (UID: \"69b4b46b-eee4-49fc-8165-b9b6d664535f\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:34:37.235184 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.234422    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69b4b46b-eee4-49fc-8165-b9b6d664535f-tmp-dir\") pod \"e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5\" (UID: \"69b4b46b-eee4-49fc-8165-b9b6d664535f\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:34:37.243798 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.243725    2566 generic.go:356] "Generic (PLEG): container finished" podID="cf39e7e4-3643-4489-887c-22ce83a4f5f1" containerID="9561441fd410c1aa96c795f5d10324dd9144d5c7b7d1eeaec76377736d1577c5" exitCode=0
Sep 18 13:34:37.243968 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.243807    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt" event={"ID":"cf39e7e4-3643-4489-887c-22ce83a4f5f1","Type":"ContainerDied","Data":"9561441fd410c1aa96c795f5d10324dd9144d5c7b7d1eeaec76377736d1577c5"}
Sep 18 13:34:37.243968 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.243911    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt" event={"ID":"cf39e7e4-3643-4489-887c-22ce83a4f5f1","Type":"ContainerDied","Data":"ce7d792d1a2c3ee2a8596fb5a5f82998f1b6ebe6f8a9fa57b2917a3c8d73f102"}
Sep 18 13:34:37.243968 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.243821    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"
Sep 18 13:34:37.243968 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.243942    2566 scope.go:117] "RemoveContainer" containerID="9561441fd410c1aa96c795f5d10324dd9144d5c7b7d1eeaec76377736d1577c5"
Sep 18 13:34:37.253454 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.253432    2566 scope.go:117] "RemoveContainer" containerID="5c10c690e743e773e7875cd4a36c638b539af7972694df02372c1da2ca5ea81f"
Sep 18 13:34:37.262079 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.262060    2566 scope.go:117] "RemoveContainer" containerID="9561441fd410c1aa96c795f5d10324dd9144d5c7b7d1eeaec76377736d1577c5"
Sep 18 13:34:37.262368 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:34:37.262349    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9561441fd410c1aa96c795f5d10324dd9144d5c7b7d1eeaec76377736d1577c5\": container with ID starting with 9561441fd410c1aa96c795f5d10324dd9144d5c7b7d1eeaec76377736d1577c5 not found: ID does not exist" containerID="9561441fd410c1aa96c795f5d10324dd9144d5c7b7d1eeaec76377736d1577c5"
Sep 18 13:34:37.262449 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.262382    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9561441fd410c1aa96c795f5d10324dd9144d5c7b7d1eeaec76377736d1577c5"} err="failed to get container status \"9561441fd410c1aa96c795f5d10324dd9144d5c7b7d1eeaec76377736d1577c5\": rpc error: code = NotFound desc = could not find container \"9561441fd410c1aa96c795f5d10324dd9144d5c7b7d1eeaec76377736d1577c5\": container with ID starting with 9561441fd410c1aa96c795f5d10324dd9144d5c7b7d1eeaec76377736d1577c5 not found: ID does not exist"
Sep 18 13:34:37.262449 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.262407    2566 scope.go:117] "RemoveContainer" containerID="5c10c690e743e773e7875cd4a36c638b539af7972694df02372c1da2ca5ea81f"
Sep 18 13:34:37.262714 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:34:37.262697    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c10c690e743e773e7875cd4a36c638b539af7972694df02372c1da2ca5ea81f\": container with ID starting with 5c10c690e743e773e7875cd4a36c638b539af7972694df02372c1da2ca5ea81f not found: ID does not exist" containerID="5c10c690e743e773e7875cd4a36c638b539af7972694df02372c1da2ca5ea81f"
Sep 18 13:34:37.262786 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.262720    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c10c690e743e773e7875cd4a36c638b539af7972694df02372c1da2ca5ea81f"} err="failed to get container status \"5c10c690e743e773e7875cd4a36c638b539af7972694df02372c1da2ca5ea81f\": rpc error: code = NotFound desc = could not find container \"5c10c690e743e773e7875cd4a36c638b539af7972694df02372c1da2ca5ea81f\": container with ID starting with 5c10c690e743e773e7875cd4a36c638b539af7972694df02372c1da2ca5ea81f not found: ID does not exist"
Sep 18 13:34:37.266376 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.266351    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"]
Sep 18 13:34:37.270306 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.270280    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-bb3ce27c-openshift-default-99748d596-mrgdt"]
Sep 18 13:34:37.335577 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.335533    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69b4b46b-eee4-49fc-8165-b9b6d664535f-tls-certs\") pod \"e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5\" (UID: \"69b4b46b-eee4-49fc-8165-b9b6d664535f\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:34:37.335808 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.335591    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxxsn\" (UniqueName: \"kubernetes.io/projected/69b4b46b-eee4-49fc-8165-b9b6d664535f-kube-api-access-rxxsn\") pod \"e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5\" (UID: \"69b4b46b-eee4-49fc-8165-b9b6d664535f\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:34:37.335808 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.335657    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69b4b46b-eee4-49fc-8165-b9b6d664535f-home\") pod \"e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5\" (UID: \"69b4b46b-eee4-49fc-8165-b9b6d664535f\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:34:37.335808 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.335689    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69b4b46b-eee4-49fc-8165-b9b6d664535f-dshm\") pod \"e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5\" (UID: \"69b4b46b-eee4-49fc-8165-b9b6d664535f\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:34:37.335808 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.335748    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69b4b46b-eee4-49fc-8165-b9b6d664535f-model-cache\") pod \"e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5\" (UID: \"69b4b46b-eee4-49fc-8165-b9b6d664535f\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:34:37.335808 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.335777    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69b4b46b-eee4-49fc-8165-b9b6d664535f-tmp-dir\") pod \"e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5\" (UID: \"69b4b46b-eee4-49fc-8165-b9b6d664535f\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:34:37.336250 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.336219    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69b4b46b-eee4-49fc-8165-b9b6d664535f-model-cache\") pod \"e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5\" (UID: \"69b4b46b-eee4-49fc-8165-b9b6d664535f\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:34:37.336380 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.336281    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69b4b46b-eee4-49fc-8165-b9b6d664535f-tmp-dir\") pod \"e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5\" (UID: \"69b4b46b-eee4-49fc-8165-b9b6d664535f\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:34:37.336380 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.336301    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69b4b46b-eee4-49fc-8165-b9b6d664535f-home\") pod \"e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5\" (UID: \"69b4b46b-eee4-49fc-8165-b9b6d664535f\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:34:37.338123 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.338103    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69b4b46b-eee4-49fc-8165-b9b6d664535f-dshm\") pod \"e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5\" (UID: \"69b4b46b-eee4-49fc-8165-b9b6d664535f\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:34:37.338302 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.338285    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69b4b46b-eee4-49fc-8165-b9b6d664535f-tls-certs\") pod \"e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5\" (UID: \"69b4b46b-eee4-49fc-8165-b9b6d664535f\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:34:37.344185 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.344166    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxxsn\" (UniqueName: \"kubernetes.io/projected/69b4b46b-eee4-49fc-8165-b9b6d664535f-kube-api-access-rxxsn\") pod \"e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5\" (UID: \"69b4b46b-eee4-49fc-8165-b9b6d664535f\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:34:37.426408 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.426314    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:34:37.701160 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:37.701132    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"]
Sep 18 13:34:37.702726 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:34:37.702695    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69b4b46b_eee4_49fc_8165_b9b6d664535f.slice/crio-b72acb8cbaafe2f9cb218b01db11cf6e3360ed0827e7b99bcd2886f7b987aca7 WatchSource:0}: Error finding container b72acb8cbaafe2f9cb218b01db11cf6e3360ed0827e7b99bcd2886f7b987aca7: Status 404 returned error can't find the container with id b72acb8cbaafe2f9cb218b01db11cf6e3360ed0827e7b99bcd2886f7b987aca7
Sep 18 13:34:38.076951 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:38.076863    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf39e7e4-3643-4489-887c-22ce83a4f5f1" path="/var/lib/kubelet/pods/cf39e7e4-3643-4489-887c-22ce83a4f5f1/volumes"
Sep 18 13:34:38.251980 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:38.251941    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5" event={"ID":"69b4b46b-eee4-49fc-8165-b9b6d664535f","Type":"ContainerStarted","Data":"bcef7297356f46ba3106bd00d6a5e9cacb0f19593d21a938b18437ffdf5ee2a7"}
Sep 18 13:34:38.251980 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:38.251984    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5" event={"ID":"69b4b46b-eee4-49fc-8165-b9b6d664535f","Type":"ContainerStarted","Data":"b72acb8cbaafe2f9cb218b01db11cf6e3360ed0827e7b99bcd2886f7b987aca7"}
Sep 18 13:34:38.270784 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:38.270731    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5" podStartSLOduration=1.270714789 podStartE2EDuration="1.270714789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:34:38.268511497 +0000 UTC m=+1186.703387903" watchObservedRunningTime="2026-09-18 13:34:38.270714789 +0000 UTC m=+1186.705591193"
Sep 18 13:34:46.727079 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.727043    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"]
Sep 18 13:34:46.727523 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.727510    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf39e7e4-3643-4489-887c-22ce83a4f5f1" containerName="istio-proxy"
Sep 18 13:34:46.730519 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.730498    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:34:46.733533 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.733501    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-951469\"/\"default-dockercfg-lxfdl\""
Sep 18 13:34:46.733667 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.733506    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-951469\"/\"kube-root-ca.crt\""
Sep 18 13:34:46.733667 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.733543    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-951469\"/\"openshift-service-ca.crt\""
Sep 18 13:34:46.733667 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.733511    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-951469\"/\"distinct-w0-951469-kserve-self-signed-certs\""
Sep 18 13:34:46.755011 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.754979    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"]
Sep 18 13:34:46.830654 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.830586    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fc70eb-512e-42d0-89ef-f91eb7e00e99-tmp-dir\") pod \"distinct-w0-951469-kserve-dbbc8fb6f-q49lh\" (UID: \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\") " pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:34:46.830811 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.830662    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50fc70eb-512e-42d0-89ef-f91eb7e00e99-home\") pod \"distinct-w0-951469-kserve-dbbc8fb6f-q49lh\" (UID: \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\") " pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:34:46.830811 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.830683    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7phkd\" (UniqueName: \"kubernetes.io/projected/50fc70eb-512e-42d0-89ef-f91eb7e00e99-kube-api-access-7phkd\") pod \"distinct-w0-951469-kserve-dbbc8fb6f-q49lh\" (UID: \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\") " pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:34:46.830811 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.830706    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50fc70eb-512e-42d0-89ef-f91eb7e00e99-dshm\") pod \"distinct-w0-951469-kserve-dbbc8fb6f-q49lh\" (UID: \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\") " pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:34:46.830811 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.830786    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50fc70eb-512e-42d0-89ef-f91eb7e00e99-tls-certs\") pod \"distinct-w0-951469-kserve-dbbc8fb6f-q49lh\" (UID: \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\") " pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:34:46.830991 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.830836    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50fc70eb-512e-42d0-89ef-f91eb7e00e99-model-cache\") pod \"distinct-w0-951469-kserve-dbbc8fb6f-q49lh\" (UID: \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\") " pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:34:46.932251 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.932219    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fc70eb-512e-42d0-89ef-f91eb7e00e99-tmp-dir\") pod \"distinct-w0-951469-kserve-dbbc8fb6f-q49lh\" (UID: \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\") " pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:34:46.932251 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.932263    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50fc70eb-512e-42d0-89ef-f91eb7e00e99-home\") pod \"distinct-w0-951469-kserve-dbbc8fb6f-q49lh\" (UID: \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\") " pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:34:46.932460 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.932283    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7phkd\" (UniqueName: \"kubernetes.io/projected/50fc70eb-512e-42d0-89ef-f91eb7e00e99-kube-api-access-7phkd\") pod \"distinct-w0-951469-kserve-dbbc8fb6f-q49lh\" (UID: \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\") " pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:34:46.932460 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.932302    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50fc70eb-512e-42d0-89ef-f91eb7e00e99-dshm\") pod \"distinct-w0-951469-kserve-dbbc8fb6f-q49lh\" (UID: \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\") " pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:34:46.932460 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.932347    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50fc70eb-512e-42d0-89ef-f91eb7e00e99-tls-certs\") pod \"distinct-w0-951469-kserve-dbbc8fb6f-q49lh\" (UID: \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\") " pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:34:46.932460 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.932371    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50fc70eb-512e-42d0-89ef-f91eb7e00e99-model-cache\") pod \"distinct-w0-951469-kserve-dbbc8fb6f-q49lh\" (UID: \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\") " pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:34:46.932726 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.932690    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fc70eb-512e-42d0-89ef-f91eb7e00e99-tmp-dir\") pod \"distinct-w0-951469-kserve-dbbc8fb6f-q49lh\" (UID: \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\") " pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:34:46.932726 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.932718    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50fc70eb-512e-42d0-89ef-f91eb7e00e99-home\") pod \"distinct-w0-951469-kserve-dbbc8fb6f-q49lh\" (UID: \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\") " pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:34:46.932837 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.932815    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50fc70eb-512e-42d0-89ef-f91eb7e00e99-model-cache\") pod \"distinct-w0-951469-kserve-dbbc8fb6f-q49lh\" (UID: \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\") " pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:34:46.934642 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.934623    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50fc70eb-512e-42d0-89ef-f91eb7e00e99-dshm\") pod \"distinct-w0-951469-kserve-dbbc8fb6f-q49lh\" (UID: \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\") " pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:34:46.934930 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.934910    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50fc70eb-512e-42d0-89ef-f91eb7e00e99-tls-certs\") pod \"distinct-w0-951469-kserve-dbbc8fb6f-q49lh\" (UID: \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\") " pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:34:46.944940 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:46.944915    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7phkd\" (UniqueName: \"kubernetes.io/projected/50fc70eb-512e-42d0-89ef-f91eb7e00e99-kube-api-access-7phkd\") pod \"distinct-w0-951469-kserve-dbbc8fb6f-q49lh\" (UID: \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\") " pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:34:47.039420 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.039329    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:34:47.350253 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.350080    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"]
Sep 18 13:34:47.353065 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:34:47.353040    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50fc70eb_512e_42d0_89ef_f91eb7e00e99.slice/crio-a6fea2103acd53c11b63bb22487e9fc623f638754a318d81935fadadf4c4c0ce WatchSource:0}: Error finding container a6fea2103acd53c11b63bb22487e9fc623f638754a318d81935fadadf4c4c0ce: Status 404 returned error can't find the container with id a6fea2103acd53c11b63bb22487e9fc623f638754a318d81935fadadf4c4c0ce
Sep 18 13:34:47.428003 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.427282    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:34:47.428179 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.428035    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:34:47.444855 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.444825    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:34:47.552170 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.552138    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"]
Sep 18 13:34:47.559502 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.559468    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.566329 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.566300    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-b2d5da-openshift-default-dockercfg-mh5nz\""
Sep 18 13:34:47.585014 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.584986    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"]
Sep 18 13:34:47.640521 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.640443    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-credential-socket\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.640521 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.640482    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17414e82-b0f6-4822-9eda-0b7d0a567925-istiod-ca-cert\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.640743 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.640522    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17414e82-b0f6-4822-9eda-0b7d0a567925-istio-podinfo\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.640743 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.640648    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-workload-certs\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.640743 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.640731    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gf58\" (UniqueName: \"kubernetes.io/projected/17414e82-b0f6-4822-9eda-0b7d0a567925-kube-api-access-6gf58\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.640846 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.640765    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-istio-data\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.640846 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.640820    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17414e82-b0f6-4822-9eda-0b7d0a567925-istio-token\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.640926 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.640845    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-istio-envoy\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.640926 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.640884    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-workload-socket\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.741824 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.741775    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gf58\" (UniqueName: \"kubernetes.io/projected/17414e82-b0f6-4822-9eda-0b7d0a567925-kube-api-access-6gf58\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.741824 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.741833    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-istio-data\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.742343 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.741856    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17414e82-b0f6-4822-9eda-0b7d0a567925-istio-token\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.742343 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.741874    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-istio-envoy\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.742343 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.741900    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-workload-socket\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.742343 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.741944    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-credential-socket\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.742343 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.741972    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17414e82-b0f6-4822-9eda-0b7d0a567925-istiod-ca-cert\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.742343 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.742014    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17414e82-b0f6-4822-9eda-0b7d0a567925-istio-podinfo\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.742343 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.742068    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-workload-certs\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.742758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.742348    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-istio-data\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.742758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.742403    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-workload-socket\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.742758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.742499    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-credential-socket\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.742758 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.742680    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-workload-certs\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.742893 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.742800    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17414e82-b0f6-4822-9eda-0b7d0a567925-istiod-ca-cert\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.744394 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.744370    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-istio-envoy\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.744719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.744703    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17414e82-b0f6-4822-9eda-0b7d0a567925-istio-podinfo\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.756289 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.756264    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17414e82-b0f6-4822-9eda-0b7d0a567925-istio-token\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.756414 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.756399    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gf58\" (UniqueName: \"kubernetes.io/projected/17414e82-b0f6-4822-9eda-0b7d0a567925-kube-api-access-6gf58\") pod \"e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") " pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:47.870782 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:47.870741    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:48.152440 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:48.152389    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"]
Sep 18 13:34:48.154687 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:34:48.154661    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17414e82_b0f6_4822_9eda_0b7d0a567925.slice/crio-61abd110a3688093638c3870b5494b276970effcf010fcd1ccb8fcd723e69fe3 WatchSource:0}: Error finding container 61abd110a3688093638c3870b5494b276970effcf010fcd1ccb8fcd723e69fe3: Status 404 returned error can't find the container with id 61abd110a3688093638c3870b5494b276970effcf010fcd1ccb8fcd723e69fe3
Sep 18 13:34:48.157972 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:48.157932    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:34:48.158084 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:48.158019    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:34:48.158084 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:48.158062    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:34:48.312503 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:48.312464    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg" event={"ID":"17414e82-b0f6-4822-9eda-0b7d0a567925","Type":"ContainerStarted","Data":"2f279ca03a1c427ffb0d5114a54207e46ae3125c9bbc61a01b35df191e67052c"}
Sep 18 13:34:48.312728 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:48.312535    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg" event={"ID":"17414e82-b0f6-4822-9eda-0b7d0a567925","Type":"ContainerStarted","Data":"61abd110a3688093638c3870b5494b276970effcf010fcd1ccb8fcd723e69fe3"}
Sep 18 13:34:48.314281 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:48.314252    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh" event={"ID":"50fc70eb-512e-42d0-89ef-f91eb7e00e99","Type":"ContainerStarted","Data":"260ba8d000110c641f78d83592e24d2f7cd829c85f9d6925887b577ea121b864"}
Sep 18 13:34:48.314412 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:48.314293    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh" event={"ID":"50fc70eb-512e-42d0-89ef-f91eb7e00e99","Type":"ContainerStarted","Data":"a6fea2103acd53c11b63bb22487e9fc623f638754a318d81935fadadf4c4c0ce"}
Sep 18 13:34:48.327302 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:48.327273    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:34:48.342166 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:48.342096    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg" podStartSLOduration=1.342071841 podStartE2EDuration="1.342071841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:34:48.336739775 +0000 UTC m=+1196.771616182" watchObservedRunningTime="2026-09-18 13:34:48.342071841 +0000 UTC m=+1196.776948241"
Sep 18 13:34:48.361411 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:48.361323    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh" podStartSLOduration=2.361299281 podStartE2EDuration="2.361299281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:34:48.361165773 +0000 UTC m=+1196.796042177" watchObservedRunningTime="2026-09-18 13:34:48.361299281 +0000 UTC m=+1196.796175685"
Sep 18 13:34:48.871660 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:48.871623    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:49.876238 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:49.876207    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:50.325831 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:50.325748    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:50.326786 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:50.326758    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:34:52.050508 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:52.050481    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k86vn_55e33e14-3f7a-4793-a724-9fd5022740d7/console-operator/1.log"
Sep 18 13:34:52.055110 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:52.055084    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k86vn_55e33e14-3f7a-4793-a724-9fd5022740d7/console-operator/1.log"
Sep 18 13:34:57.040076 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:57.040031    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:34:57.040545 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:57.040333    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:34:57.052887 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:57.052866    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:34:57.374889 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:34:57.374859    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:35:21.399222 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.399186    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"]
Sep 18 13:35:21.402889 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.402866    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.406044 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.406021    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-92b54f-openshift-default-dockercfg-6m8hz\""
Sep 18 13:35:21.427565 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.427535    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"]
Sep 18 13:35:21.495060 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.494507    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istio-envoy\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.495060 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.494558    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istio-podinfo\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.495060 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.494696    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-workload-certs\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.495060 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.494738    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istiod-ca-cert\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.495060 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.494787    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istio-data\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.495060 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.494815    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-credential-socket\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.495060 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.494839    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istio-token\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.495060 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.494865    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-workload-socket\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.495060 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.494918    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnmkc\" (UniqueName: \"kubernetes.io/projected/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-kube-api-access-hnmkc\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.596426 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.596388    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-workload-certs\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.596668 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.596435    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istiod-ca-cert\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.596668 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.596561    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istio-data\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.596668 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.596629    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-credential-socket\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.596668 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.596653    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istio-token\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.596925 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.596785    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-workload-socket\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.596925 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.596871    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnmkc\" (UniqueName: \"kubernetes.io/projected/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-kube-api-access-hnmkc\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.596925 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.596909    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-workload-certs\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.597091 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.596955    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istio-data\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.597091 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.597010    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istio-envoy\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.597091 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.597036    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-credential-socket\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.597091 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.597051    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istio-podinfo\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.597377 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.597356    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istiod-ca-cert\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.597633 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.597588    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-workload-socket\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.599225 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.599204    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istio-envoy\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.599434 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.599418    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istio-podinfo\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.609574 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.609551    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istio-token\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.611114 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.611096    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnmkc\" (UniqueName: \"kubernetes.io/projected/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-kube-api-access-hnmkc\") pod \"e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") " pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:21.714047 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:21.713933    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:22.022857 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:22.022830    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"]
Sep 18 13:35:22.024302 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:35:22.024275    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9244fde_4e58_4ee0_a4ac_17827a4ac2dd.slice/crio-83bfc346312fadf9e9020b3b04b8a2a78762cc8c46bb38e93916a6b46f84f08e WatchSource:0}: Error finding container 83bfc346312fadf9e9020b3b04b8a2a78762cc8c46bb38e93916a6b46f84f08e: Status 404 returned error can't find the container with id 83bfc346312fadf9e9020b3b04b8a2a78762cc8c46bb38e93916a6b46f84f08e
Sep 18 13:35:22.029928 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:22.029885    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:35:22.030051 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:22.030005    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:35:22.030120 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:22.030051    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:35:22.514305 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:22.514268    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l" event={"ID":"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd","Type":"ContainerStarted","Data":"33f3e61a0973b8510d06445884f51cdec4904fc66fb653ebe901dc0cf72830d6"}
Sep 18 13:35:22.514784 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:22.514328    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l" event={"ID":"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd","Type":"ContainerStarted","Data":"83bfc346312fadf9e9020b3b04b8a2a78762cc8c46bb38e93916a6b46f84f08e"}
Sep 18 13:35:22.543129 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:22.543064    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l" podStartSLOduration=1.5430406620000001 podStartE2EDuration="1.543040662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:35:22.539029591 +0000 UTC m=+1230.973905995" watchObservedRunningTime="2026-09-18 13:35:22.543040662 +0000 UTC m=+1230.977917067"
Sep 18 13:35:22.714292 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:22.714245    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:23.721425 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:23.721382    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:24.526563 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:24.526526    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:24.527498 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:24.527474    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:26.539513 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:26.539427    2566 generic.go:356] "Generic (PLEG): container finished" podID="17414e82-b0f6-4822-9eda-0b7d0a567925" containerID="2f279ca03a1c427ffb0d5114a54207e46ae3125c9bbc61a01b35df191e67052c" exitCode=0
Sep 18 13:35:26.539513 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:26.539499    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg" event={"ID":"17414e82-b0f6-4822-9eda-0b7d0a567925","Type":"ContainerDied","Data":"2f279ca03a1c427ffb0d5114a54207e46ae3125c9bbc61a01b35df191e67052c"}
Sep 18 13:35:26.540087 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:26.540072    2566 scope.go:117] "RemoveContainer" containerID="2f279ca03a1c427ffb0d5114a54207e46ae3125c9bbc61a01b35df191e67052c"
Sep 18 13:35:26.541166 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:26.541142    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:35:26.541219 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:26.541203    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:35:26.541255 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:26.541246    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:35:27.552438 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:27.552383    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg" event={"ID":"17414e82-b0f6-4822-9eda-0b7d0a567925","Type":"ContainerStarted","Data":"3c6f17d69627c95bf7e0c09da7be5983375d400a8e4cf0c06f9b3a41ab6e46de"}
Sep 18 13:35:27.871201 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:27.871153    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:35:27.871515 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:27.871482    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 18 13:35:27.871675 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:27.871539    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg" podUID="17414e82-b0f6-4822-9eda-0b7d0a567925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 18 13:35:28.871160 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:28.871128    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 18 13:35:28.871688 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:28.871181    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg" podUID="17414e82-b0f6-4822-9eda-0b7d0a567925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 18 13:35:29.872080 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:29.872043    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 18 13:35:29.872493 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:29.872115    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg" podUID="17414e82-b0f6-4822-9eda-0b7d0a567925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 18 13:35:30.871152 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:30.871115    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 18 13:35:30.871347 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:30.871184    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg" podUID="17414e82-b0f6-4822-9eda-0b7d0a567925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 18 13:35:31.871558 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:31.871514    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 18 13:35:31.871977 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:31.871618    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg" podUID="17414e82-b0f6-4822-9eda-0b7d0a567925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 18 13:35:32.871309 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:32.871271    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 18 13:35:32.871524 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:32.871340    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg" podUID="17414e82-b0f6-4822-9eda-0b7d0a567925" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 18 13:35:33.876770 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:33.876736    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:35:33.877245 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:33.877144    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:35:33.877948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:33.877916    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:35:52.137546 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:52.137464    2566 scope.go:117] "RemoveContainer" containerID="12b0de2158e424c88d5da600c27221e7ddd0ecf4cb754472ed7b79cf4dcf141c"
Sep 18 13:35:57.731207 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:57.731095    2566 generic.go:356] "Generic (PLEG): container finished" podID="a9244fde-4e58-4ee0-a4ac-17827a4ac2dd" containerID="33f3e61a0973b8510d06445884f51cdec4904fc66fb653ebe901dc0cf72830d6" exitCode=0
Sep 18 13:35:57.731207 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:57.731152    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l" event={"ID":"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd","Type":"ContainerDied","Data":"33f3e61a0973b8510d06445884f51cdec4904fc66fb653ebe901dc0cf72830d6"}
Sep 18 13:35:57.731816 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:57.731647    2566 scope.go:117] "RemoveContainer" containerID="33f3e61a0973b8510d06445884f51cdec4904fc66fb653ebe901dc0cf72830d6"
Sep 18 13:35:57.733059 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:57.733024    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:35:57.733174 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:57.733095    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:35:57.733174 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:57.733124    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:35:58.739345 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:58.739292    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l" event={"ID":"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd","Type":"ContainerStarted","Data":"fc4f4b7e012d831333391208e1f2fc6c2e464b788888cc2e0a44a7cd85ab88ba"}
Sep 18 13:35:59.714896 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:59.714860    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:35:59.715160 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:59.715135    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 18 13:35:59.715220 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:35:59.715198    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l" podUID="a9244fde-4e58-4ee0-a4ac-17827a4ac2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 18 13:36:00.714575 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:00.714539    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 18 13:36:00.715084 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:00.714619    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l" podUID="a9244fde-4e58-4ee0-a4ac-17827a4ac2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 18 13:36:01.668424 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.668378    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"]
Sep 18 13:36:01.671809 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.671783    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:01.681769 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.681747    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-b81e47\"/\"kube-root-ca.crt\""
Sep 18 13:36:01.682272 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.682256    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-b81e47\"/\"openshift-service-ca.crt\""
Sep 18 13:36:01.682357 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.682282    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-b81e47\"/\"default-dockercfg-rf7bq\""
Sep 18 13:36:01.682693 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.682679    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-b81e47\"/\"noten-model-b81e47-kserve-self-signed-certs\""
Sep 18 13:36:01.694198 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.694175    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"]
Sep 18 13:36:01.715149 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.715114    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 18 13:36:01.715488 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.715198    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l" podUID="a9244fde-4e58-4ee0-a4ac-17827a4ac2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 18 13:36:01.720965 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.720936    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-692hq\" (UniqueName: \"kubernetes.io/projected/1d3d0335-76e3-4912-9def-f50f888bc0cb-kube-api-access-692hq\") pod \"noten-model-b81e47-kserve-86c9b8f9ff-86dxw\" (UID: \"1d3d0335-76e3-4912-9def-f50f888bc0cb\") " pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:01.721100 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.720982    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d3d0335-76e3-4912-9def-f50f888bc0cb-dshm\") pod \"noten-model-b81e47-kserve-86c9b8f9ff-86dxw\" (UID: \"1d3d0335-76e3-4912-9def-f50f888bc0cb\") " pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:01.721100 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.721002    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3d0335-76e3-4912-9def-f50f888bc0cb-tmp-dir\") pod \"noten-model-b81e47-kserve-86c9b8f9ff-86dxw\" (UID: \"1d3d0335-76e3-4912-9def-f50f888bc0cb\") " pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:01.721100 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.721055    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d3d0335-76e3-4912-9def-f50f888bc0cb-model-cache\") pod \"noten-model-b81e47-kserve-86c9b8f9ff-86dxw\" (UID: \"1d3d0335-76e3-4912-9def-f50f888bc0cb\") " pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:01.721100 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.721095    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d3d0335-76e3-4912-9def-f50f888bc0cb-tls-certs\") pod \"noten-model-b81e47-kserve-86c9b8f9ff-86dxw\" (UID: \"1d3d0335-76e3-4912-9def-f50f888bc0cb\") " pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:01.721263 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.721143    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d3d0335-76e3-4912-9def-f50f888bc0cb-home\") pod \"noten-model-b81e47-kserve-86c9b8f9ff-86dxw\" (UID: \"1d3d0335-76e3-4912-9def-f50f888bc0cb\") " pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:01.822442 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.822409    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d3d0335-76e3-4912-9def-f50f888bc0cb-model-cache\") pod \"noten-model-b81e47-kserve-86c9b8f9ff-86dxw\" (UID: \"1d3d0335-76e3-4912-9def-f50f888bc0cb\") " pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:01.822693 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.822472    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d3d0335-76e3-4912-9def-f50f888bc0cb-tls-certs\") pod \"noten-model-b81e47-kserve-86c9b8f9ff-86dxw\" (UID: \"1d3d0335-76e3-4912-9def-f50f888bc0cb\") " pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:01.822693 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.822552    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d3d0335-76e3-4912-9def-f50f888bc0cb-home\") pod \"noten-model-b81e47-kserve-86c9b8f9ff-86dxw\" (UID: \"1d3d0335-76e3-4912-9def-f50f888bc0cb\") " pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:01.822820 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.822691    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-692hq\" (UniqueName: \"kubernetes.io/projected/1d3d0335-76e3-4912-9def-f50f888bc0cb-kube-api-access-692hq\") pod \"noten-model-b81e47-kserve-86c9b8f9ff-86dxw\" (UID: \"1d3d0335-76e3-4912-9def-f50f888bc0cb\") " pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:01.822820 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.822744    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d3d0335-76e3-4912-9def-f50f888bc0cb-dshm\") pod \"noten-model-b81e47-kserve-86c9b8f9ff-86dxw\" (UID: \"1d3d0335-76e3-4912-9def-f50f888bc0cb\") " pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:01.822820 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.822767    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3d0335-76e3-4912-9def-f50f888bc0cb-tmp-dir\") pod \"noten-model-b81e47-kserve-86c9b8f9ff-86dxw\" (UID: \"1d3d0335-76e3-4912-9def-f50f888bc0cb\") " pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:01.822974 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.822901    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d3d0335-76e3-4912-9def-f50f888bc0cb-model-cache\") pod \"noten-model-b81e47-kserve-86c9b8f9ff-86dxw\" (UID: \"1d3d0335-76e3-4912-9def-f50f888bc0cb\") " pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:01.822974 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.822932    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d3d0335-76e3-4912-9def-f50f888bc0cb-home\") pod \"noten-model-b81e47-kserve-86c9b8f9ff-86dxw\" (UID: \"1d3d0335-76e3-4912-9def-f50f888bc0cb\") " pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:01.823140 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.823121    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3d0335-76e3-4912-9def-f50f888bc0cb-tmp-dir\") pod \"noten-model-b81e47-kserve-86c9b8f9ff-86dxw\" (UID: \"1d3d0335-76e3-4912-9def-f50f888bc0cb\") " pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:01.825001 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.824972    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d3d0335-76e3-4912-9def-f50f888bc0cb-dshm\") pod \"noten-model-b81e47-kserve-86c9b8f9ff-86dxw\" (UID: \"1d3d0335-76e3-4912-9def-f50f888bc0cb\") " pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:01.825001 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.824995    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d3d0335-76e3-4912-9def-f50f888bc0cb-tls-certs\") pod \"noten-model-b81e47-kserve-86c9b8f9ff-86dxw\" (UID: \"1d3d0335-76e3-4912-9def-f50f888bc0cb\") " pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:01.868627 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.865476    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-692hq\" (UniqueName: \"kubernetes.io/projected/1d3d0335-76e3-4912-9def-f50f888bc0cb-kube-api-access-692hq\") pod \"noten-model-b81e47-kserve-86c9b8f9ff-86dxw\" (UID: \"1d3d0335-76e3-4912-9def-f50f888bc0cb\") " pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:01.982029 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:01.981909    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:02.310740 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:36:02.310708    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d3d0335_76e3_4912_9def_f50f888bc0cb.slice/crio-4f0b5d02baa62b63ffecfeb608b35f239ab87a570b2440f79b742afca5bd3b3e WatchSource:0}: Error finding container 4f0b5d02baa62b63ffecfeb608b35f239ab87a570b2440f79b742afca5bd3b3e: Status 404 returned error can't find the container with id 4f0b5d02baa62b63ffecfeb608b35f239ab87a570b2440f79b742afca5bd3b3e
Sep 18 13:36:02.310852 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:02.310779    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"]
Sep 18 13:36:02.714583 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:02.714539    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 18 13:36:02.714835 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:02.714648    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l" podUID="a9244fde-4e58-4ee0-a4ac-17827a4ac2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 18 13:36:02.784671 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:02.784631    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw" event={"ID":"1d3d0335-76e3-4912-9def-f50f888bc0cb","Type":"ContainerStarted","Data":"89c930af069401315cfe60927385d752d29b8aa7e19eac12c29d72abd65fa0fc"}
Sep 18 13:36:02.785076 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:02.784682    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw" event={"ID":"1d3d0335-76e3-4912-9def-f50f888bc0cb","Type":"ContainerStarted","Data":"4f0b5d02baa62b63ffecfeb608b35f239ab87a570b2440f79b742afca5bd3b3e"}
Sep 18 13:36:02.835498 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:02.835447    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw" podStartSLOduration=1.8354299969999999 podStartE2EDuration="1.835429997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:36:02.833376048 +0000 UTC m=+1271.268252454" watchObservedRunningTime="2026-09-18 13:36:02.835429997 +0000 UTC m=+1271.270306443"
Sep 18 13:36:03.147132 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:03.147071    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34aeca-mvlb9"]
Sep 18 13:36:03.150881 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:03.150853    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34aeca-mvlb9"
Sep 18 13:36:03.160814 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:03.160779    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nh2cd\""
Sep 18 13:36:03.161279 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:03.161233    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:36:03.182316 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:03.182286    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34aeca-mvlb9"]
Sep 18 13:36:03.236781 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:03.236749    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cae7ef7-04f9-4cb8-b9ef-b539f20f41f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-34aeca-mvlb9\" (UID: \"7cae7ef7-04f9-4cb8-b9ef-b539f20f41f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34aeca-mvlb9"
Sep 18 13:36:03.337682 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:03.337636    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cae7ef7-04f9-4cb8-b9ef-b539f20f41f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-34aeca-mvlb9\" (UID: \"7cae7ef7-04f9-4cb8-b9ef-b539f20f41f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34aeca-mvlb9"
Sep 18 13:36:03.338534 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:03.338499    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cae7ef7-04f9-4cb8-b9ef-b539f20f41f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-34aeca-mvlb9\" (UID: \"7cae7ef7-04f9-4cb8-b9ef-b539f20f41f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34aeca-mvlb9"
Sep 18 13:36:03.468103 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:03.468013    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34aeca-mvlb9"
Sep 18 13:36:03.715429 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:03.715385    2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 18 13:36:03.715646 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:03.715448    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l" podUID="a9244fde-4e58-4ee0-a4ac-17827a4ac2dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 18 13:36:03.764537 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:03.764496    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34aeca-mvlb9"]
Sep 18 13:36:03.764837 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:36:03.764792    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cae7ef7_04f9_4cb8_b9ef_b539f20f41f7.slice/crio-bcd75a17b83f1dcfdee6f7960b13236d64eec3b22830bde829291b36cd51874d WatchSource:0}: Error finding container bcd75a17b83f1dcfdee6f7960b13236d64eec3b22830bde829291b36cd51874d: Status 404 returned error can't find the container with id bcd75a17b83f1dcfdee6f7960b13236d64eec3b22830bde829291b36cd51874d
Sep 18 13:36:03.793086 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:03.793057    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34aeca-mvlb9" event={"ID":"7cae7ef7-04f9-4cb8-b9ef-b539f20f41f7","Type":"ContainerStarted","Data":"bcd75a17b83f1dcfdee6f7960b13236d64eec3b22830bde829291b36cd51874d"}
Sep 18 13:36:04.719706 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:04.719675    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:36:04.720082 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:04.720061    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:36:04.720708 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:04.720690    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:36:04.801207 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:04.801159    2566 generic.go:356] "Generic (PLEG): container finished" podID="7cae7ef7-04f9-4cb8-b9ef-b539f20f41f7" containerID="e1fc196a542219706bd93725d7e842ed3e34b143b2472f9a82bebb5a9f5d13b3" exitCode=0
Sep 18 13:36:04.801698 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:04.801208    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34aeca-mvlb9" event={"ID":"7cae7ef7-04f9-4cb8-b9ef-b539f20f41f7","Type":"ContainerDied","Data":"e1fc196a542219706bd93725d7e842ed3e34b143b2472f9a82bebb5a9f5d13b3"}
Sep 18 13:36:05.103674 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:05.103639    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"]
Sep 18 13:36:05.936780 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:05.936754    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34aeca-mvlb9"
Sep 18 13:36:06.089589 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:06.089511    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cae7ef7-04f9-4cb8-b9ef-b539f20f41f7-maas-api-service-ca\") pod \"7cae7ef7-04f9-4cb8-b9ef-b539f20f41f7\" (UID: \"7cae7ef7-04f9-4cb8-b9ef-b539f20f41f7\") "
Sep 18 13:36:06.089931 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:06.089910    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cae7ef7-04f9-4cb8-b9ef-b539f20f41f7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7cae7ef7-04f9-4cb8-b9ef-b539f20f41f7" (UID: "7cae7ef7-04f9-4cb8-b9ef-b539f20f41f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:36:06.191399 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:06.191363    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cae7ef7-04f9-4cb8-b9ef-b539f20f41f7-maas-api-service-ca\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:06.586315 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:06.586280    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"]
Sep 18 13:36:06.814462 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:06.814429    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34aeca-mvlb9" event={"ID":"7cae7ef7-04f9-4cb8-b9ef-b539f20f41f7","Type":"ContainerDied","Data":"bcd75a17b83f1dcfdee6f7960b13236d64eec3b22830bde829291b36cd51874d"}
Sep 18 13:36:06.814462 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:06.814449    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34aeca-mvlb9"
Sep 18 13:36:06.814791 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:06.814478    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcd75a17b83f1dcfdee6f7960b13236d64eec3b22830bde829291b36cd51874d"
Sep 18 13:36:06.920946 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:06.920912    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"]
Sep 18 13:36:06.921377 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:06.921365    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cae7ef7-04f9-4cb8-b9ef-b539f20f41f7" containerName="revoke-keys"
Sep 18 13:36:06.921419 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:06.921379    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cae7ef7-04f9-4cb8-b9ef-b539f20f41f7" containerName="revoke-keys"
Sep 18 13:36:06.921454 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:06.921441    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cae7ef7-04f9-4cb8-b9ef-b539f20f41f7" containerName="revoke-keys"
Sep 18 13:36:06.923813 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:06.923796    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:06.927870 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:06.927849    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fa9905fd-openshift-default-dockercfg-4l89x\""
Sep 18 13:36:06.928534 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:36:06.928506    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" is forbidden: User \"system:node:ip-10-0-129-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-129-234.ec2.internal' and this object" podUID="3e87c7b2-c903-4be2-9039-3499f78e2cbb" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:06.982590 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:06.982557    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"]
Sep 18 13:36:07.102783 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.102744    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istio-data\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.102783 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.102787    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-credential-socket\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.103022 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.102852    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istiod-ca-cert\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.103022 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.102891    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-workload-socket\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.103022 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.102910    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istio-podinfo\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.103022 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.102932    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf725\" (UniqueName: \"kubernetes.io/projected/3e87c7b2-c903-4be2-9039-3499f78e2cbb-kube-api-access-jf725\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.103168 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.103030    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istio-envoy\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.103168 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.103094    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istio-token\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.103168 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.103139    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-workload-certs\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.203818 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.203715    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istio-envoy\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.203818 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.203824    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istio-token\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.204073 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.203871    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-workload-certs\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.204073 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.203908    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istio-data\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.204073 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.203941    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-credential-socket\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.204073 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.204030    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istiod-ca-cert\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.204291 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.204097    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-workload-socket\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.204291 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.204126    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istio-podinfo\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.204291 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.204159    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf725\" (UniqueName: \"kubernetes.io/projected/3e87c7b2-c903-4be2-9039-3499f78e2cbb-kube-api-access-jf725\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.204291 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.204268    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-workload-certs\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.204517 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.204489    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istio-data\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.204571 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.204519    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-workload-socket\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.204683 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.204661    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-credential-socket\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.204818 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.204803    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istiod-ca-cert\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.206152 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.206129    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istio-envoy\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.206507 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.206488    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istio-podinfo\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.229148 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.229104    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istio-token\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.229297 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.229219    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf725\" (UniqueName: \"kubernetes.io/projected/3e87c7b2-c903-4be2-9039-3499f78e2cbb-kube-api-access-jf725\") pod \"e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") " pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.234008 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.233976    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:07.537253 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.537226    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"]
Sep 18 13:36:07.538540 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:36:07.538509    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e87c7b2_c903_4be2_9039_3499f78e2cbb.slice/crio-d932f65bc35e33056cf106d475ff83ec462d0554d6f32035fd75a7dcd38409c4 WatchSource:0}: Error finding container d932f65bc35e33056cf106d475ff83ec462d0554d6f32035fd75a7dcd38409c4: Status 404 returned error can't find the container with id d932f65bc35e33056cf106d475ff83ec462d0554d6f32035fd75a7dcd38409c4
Sep 18 13:36:07.540808 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.540778    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:36:07.540877 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.540838    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:36:07.540877 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.540863    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:36:07.823912 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.823828    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j" event={"ID":"3e87c7b2-c903-4be2-9039-3499f78e2cbb","Type":"ContainerStarted","Data":"f1659a0bde9830086be0299d3bc16d4c61a28de557c91418319db5a7a3bcd590"}
Sep 18 13:36:07.823912 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.823885    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j" event={"ID":"3e87c7b2-c903-4be2-9039-3499f78e2cbb","Type":"ContainerStarted","Data":"d932f65bc35e33056cf106d475ff83ec462d0554d6f32035fd75a7dcd38409c4"}
Sep 18 13:36:07.854288 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:07.854220    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j" podStartSLOduration=1.854195101 podStartE2EDuration="1.854195101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:36:07.851559294 +0000 UTC m=+1276.286435697" watchObservedRunningTime="2026-09-18 13:36:07.854195101 +0000 UTC m=+1276.289071506"
Sep 18 13:36:08.234257 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:08.234221    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:08.239543 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:08.239517    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:08.829307 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:08.829273    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:08.830319 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:08.830296    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:11.982160 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:11.982122    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:11.982672 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:11.982176    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:11.994991 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:11.994962    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:12.862660 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:12.862621    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:15.229800 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:15.229760    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc"]
Sep 18 13:36:15.234550 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:15.234517    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc"
Sep 18 13:36:15.237013 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:15.236979    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fa9905fd\""
Sep 18 13:36:15.244499 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:15.244468    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc"]
Sep 18 13:36:15.293012 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:15.292978    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af436413-64b2-4248-b6ba-2ffd075df6c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc\" (UID: \"af436413-64b2-4248-b6ba-2ffd075df6c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc"
Sep 18 13:36:15.293234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:15.293026    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ptd8\" (UniqueName: \"kubernetes.io/projected/af436413-64b2-4248-b6ba-2ffd075df6c8-kube-api-access-9ptd8\") pod \"payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc\" (UID: \"af436413-64b2-4248-b6ba-2ffd075df6c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc"
Sep 18 13:36:15.394516 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:15.394471    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af436413-64b2-4248-b6ba-2ffd075df6c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc\" (UID: \"af436413-64b2-4248-b6ba-2ffd075df6c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc"
Sep 18 13:36:15.394759 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:15.394529    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ptd8\" (UniqueName: \"kubernetes.io/projected/af436413-64b2-4248-b6ba-2ffd075df6c8-kube-api-access-9ptd8\") pod \"payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc\" (UID: \"af436413-64b2-4248-b6ba-2ffd075df6c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc"
Sep 18 13:36:15.395340 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:15.395313    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af436413-64b2-4248-b6ba-2ffd075df6c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc\" (UID: \"af436413-64b2-4248-b6ba-2ffd075df6c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc"
Sep 18 13:36:15.405582 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:15.405545    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ptd8\" (UniqueName: \"kubernetes.io/projected/af436413-64b2-4248-b6ba-2ffd075df6c8-kube-api-access-9ptd8\") pod \"payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc\" (UID: \"af436413-64b2-4248-b6ba-2ffd075df6c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc"
Sep 18 13:36:15.548640 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:15.548536    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc"
Sep 18 13:36:15.852782 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:15.852650    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc"]
Sep 18 13:36:15.855189 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:36:15.855155    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf436413_64b2_4248_b6ba_2ffd075df6c8.slice/crio-c2d3611f60f18618278803edc0d9744651560d1b78189b9b345e024983844c3c WatchSource:0}: Error finding container c2d3611f60f18618278803edc0d9744651560d1b78189b9b345e024983844c3c: Status 404 returned error can't find the container with id c2d3611f60f18618278803edc0d9744651560d1b78189b9b345e024983844c3c
Sep 18 13:36:15.894947 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:15.894905    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc" event={"ID":"af436413-64b2-4248-b6ba-2ffd075df6c8","Type":"ContainerStarted","Data":"c2d3611f60f18618278803edc0d9744651560d1b78189b9b345e024983844c3c"}
Sep 18 13:36:16.905750 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:16.905707    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc" event={"ID":"af436413-64b2-4248-b6ba-2ffd075df6c8","Type":"ContainerStarted","Data":"8690c8238c15599864b6764e0c333e48baae4ba4050a5c75f7cdf63f8ab6a779"}
Sep 18 13:36:16.906148 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:16.905806    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc"
Sep 18 13:36:16.944991 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:16.944932    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc" podStartSLOduration=1.944916629 podStartE2EDuration="1.944916629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:36:16.943133488 +0000 UTC m=+1285.378009895" watchObservedRunningTime="2026-09-18 13:36:16.944916629 +0000 UTC m=+1285.379793033"
Sep 18 13:36:18.135376 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:18.135339    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34aeca-29828970-zntzb"]
Sep 18 13:36:18.139847 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:18.139821    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-34aeca-29828970-zntzb"]
Sep 18 13:36:18.757180 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:18.757144    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34aeca-mvlb9"]
Sep 18 13:36:18.762081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:18.762049    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-34aeca-mvlb9"]
Sep 18 13:36:20.077228 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.077188    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5529278f-7402-47a8-89bb-2aa69d59f6bb" path="/var/lib/kubelet/pods/5529278f-7402-47a8-89bb-2aa69d59f6bb/volumes"
Sep 18 13:36:20.077645 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.077524    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cae7ef7-04f9-4cb8-b9ef-b539f20f41f7" path="/var/lib/kubelet/pods/7cae7ef7-04f9-4cb8-b9ef-b539f20f41f7/volumes"
Sep 18 13:36:20.123520 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.123478    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg" podUID="65ed58ca-08b3-4da7-b533-0d184bd998fe" containerName="main" containerID="cri-o://a178ca5fb631b3edd72993b8a96edb019554f95362cb5aa74d966ce28d009ebf" gracePeriod=45
Sep 18 13:36:20.373556 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.373530    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:36:20.451894 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.451855    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sp7j7\" (UniqueName: \"kubernetes.io/projected/65ed58ca-08b3-4da7-b533-0d184bd998fe-kube-api-access-sp7j7\") pod \"65ed58ca-08b3-4da7-b533-0d184bd998fe\" (UID: \"65ed58ca-08b3-4da7-b533-0d184bd998fe\") "
Sep 18 13:36:20.452056 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.451905    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65ed58ca-08b3-4da7-b533-0d184bd998fe-home\") pod \"65ed58ca-08b3-4da7-b533-0d184bd998fe\" (UID: \"65ed58ca-08b3-4da7-b533-0d184bd998fe\") "
Sep 18 13:36:20.452056 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.451946    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ed58ca-08b3-4da7-b533-0d184bd998fe-dshm\") pod \"65ed58ca-08b3-4da7-b533-0d184bd998fe\" (UID: \"65ed58ca-08b3-4da7-b533-0d184bd998fe\") "
Sep 18 13:36:20.452056 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.452003    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ed58ca-08b3-4da7-b533-0d184bd998fe-model-cache\") pod \"65ed58ca-08b3-4da7-b533-0d184bd998fe\" (UID: \"65ed58ca-08b3-4da7-b533-0d184bd998fe\") "
Sep 18 13:36:20.452253 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.452086    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ed58ca-08b3-4da7-b533-0d184bd998fe-tmp-dir\") pod \"65ed58ca-08b3-4da7-b533-0d184bd998fe\" (UID: \"65ed58ca-08b3-4da7-b533-0d184bd998fe\") "
Sep 18 13:36:20.452253 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.452220    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65ed58ca-08b3-4da7-b533-0d184bd998fe-tls-certs\") pod \"65ed58ca-08b3-4da7-b533-0d184bd998fe\" (UID: \"65ed58ca-08b3-4da7-b533-0d184bd998fe\") "
Sep 18 13:36:20.452370 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.452240    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ed58ca-08b3-4da7-b533-0d184bd998fe-home" (OuterVolumeSpecName: "home") pod "65ed58ca-08b3-4da7-b533-0d184bd998fe" (UID: "65ed58ca-08b3-4da7-b533-0d184bd998fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:20.452370 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.452270    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ed58ca-08b3-4da7-b533-0d184bd998fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "65ed58ca-08b3-4da7-b533-0d184bd998fe" (UID: "65ed58ca-08b3-4da7-b533-0d184bd998fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:20.452479 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.452427    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ed58ca-08b3-4da7-b533-0d184bd998fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "65ed58ca-08b3-4da7-b533-0d184bd998fe" (UID: "65ed58ca-08b3-4da7-b533-0d184bd998fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:20.452856 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.452831    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ed58ca-08b3-4da7-b533-0d184bd998fe-tmp-dir\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:20.452856 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.452857    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65ed58ca-08b3-4da7-b533-0d184bd998fe-home\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:20.453036 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.452867    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ed58ca-08b3-4da7-b533-0d184bd998fe-model-cache\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:20.454639 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.454609    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65ed58ca-08b3-4da7-b533-0d184bd998fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "65ed58ca-08b3-4da7-b533-0d184bd998fe" (UID: "65ed58ca-08b3-4da7-b533-0d184bd998fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:36:20.454639 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.454624    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ed58ca-08b3-4da7-b533-0d184bd998fe-dshm" (OuterVolumeSpecName: "dshm") pod "65ed58ca-08b3-4da7-b533-0d184bd998fe" (UID: "65ed58ca-08b3-4da7-b533-0d184bd998fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:20.454788 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.454669    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ed58ca-08b3-4da7-b533-0d184bd998fe-kube-api-access-sp7j7" (OuterVolumeSpecName: "kube-api-access-sp7j7") pod "65ed58ca-08b3-4da7-b533-0d184bd998fe" (UID: "65ed58ca-08b3-4da7-b533-0d184bd998fe"). InnerVolumeSpecName "kube-api-access-sp7j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:36:20.554369 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.554332    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65ed58ca-08b3-4da7-b533-0d184bd998fe-tls-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:20.554369 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.554363    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp7j7\" (UniqueName: \"kubernetes.io/projected/65ed58ca-08b3-4da7-b533-0d184bd998fe-kube-api-access-sp7j7\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:20.554369 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.554372    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ed58ca-08b3-4da7-b533-0d184bd998fe-dshm\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:20.929841 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.929806    2566 generic.go:356] "Generic (PLEG): container finished" podID="65ed58ca-08b3-4da7-b533-0d184bd998fe" containerID="a178ca5fb631b3edd72993b8a96edb019554f95362cb5aa74d966ce28d009ebf" exitCode=0
Sep 18 13:36:20.930042 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.929854    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"
Sep 18 13:36:20.930042 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.929887    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg" event={"ID":"65ed58ca-08b3-4da7-b533-0d184bd998fe","Type":"ContainerDied","Data":"a178ca5fb631b3edd72993b8a96edb019554f95362cb5aa74d966ce28d009ebf"}
Sep 18 13:36:20.930042 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.929933    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg" event={"ID":"65ed58ca-08b3-4da7-b533-0d184bd998fe","Type":"ContainerDied","Data":"885eb6e21f990e7d27dcdb03bb47dbd569bb2eba2eeb4b62959d15db246f0bb1"}
Sep 18 13:36:20.930042 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.929954    2566 scope.go:117] "RemoveContainer" containerID="a178ca5fb631b3edd72993b8a96edb019554f95362cb5aa74d966ce28d009ebf"
Sep 18 13:36:20.939940 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.939919    2566 scope.go:117] "RemoveContainer" containerID="a178ca5fb631b3edd72993b8a96edb019554f95362cb5aa74d966ce28d009ebf"
Sep 18 13:36:20.940238 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:36:20.940208    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a178ca5fb631b3edd72993b8a96edb019554f95362cb5aa74d966ce28d009ebf\": container with ID starting with a178ca5fb631b3edd72993b8a96edb019554f95362cb5aa74d966ce28d009ebf not found: ID does not exist" containerID="a178ca5fb631b3edd72993b8a96edb019554f95362cb5aa74d966ce28d009ebf"
Sep 18 13:36:20.940335 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.940237    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a178ca5fb631b3edd72993b8a96edb019554f95362cb5aa74d966ce28d009ebf"} err="failed to get container status \"a178ca5fb631b3edd72993b8a96edb019554f95362cb5aa74d966ce28d009ebf\": rpc error: code = NotFound desc = could not find container \"a178ca5fb631b3edd72993b8a96edb019554f95362cb5aa74d966ce28d009ebf\": container with ID starting with a178ca5fb631b3edd72993b8a96edb019554f95362cb5aa74d966ce28d009ebf not found: ID does not exist"
Sep 18 13:36:20.956442 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.956413    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"]
Sep 18 13:36:20.963208 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:20.963179    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-34aeca/test-model-34aeca-kserve-55f79d4f87-c68tg"]
Sep 18 13:36:21.605406 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:21.605356    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc" podUID="6cab385d-33de-463b-8886-ab159b73701b" containerName="main" containerID="cri-o://1ad699a0170b9eeda37c8af4ae3c9021e656c732667ca9ae16336d0e402a6a30" gracePeriod=45
Sep 18 13:36:21.867963 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:21.867900    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:36:21.937710 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:21.937677    2566 generic.go:356] "Generic (PLEG): container finished" podID="6cab385d-33de-463b-8886-ab159b73701b" containerID="1ad699a0170b9eeda37c8af4ae3c9021e656c732667ca9ae16336d0e402a6a30" exitCode=0
Sep 18 13:36:21.937892 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:21.937725    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"
Sep 18 13:36:21.937892 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:21.937761    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc" event={"ID":"6cab385d-33de-463b-8886-ab159b73701b","Type":"ContainerDied","Data":"1ad699a0170b9eeda37c8af4ae3c9021e656c732667ca9ae16336d0e402a6a30"}
Sep 18 13:36:21.937892 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:21.937804    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc" event={"ID":"6cab385d-33de-463b-8886-ab159b73701b","Type":"ContainerDied","Data":"e0e4e148fe45cafca5c4eeba158ae6bc528b8f87d7b2f9bb3dce82a8d2761609"}
Sep 18 13:36:21.937892 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:21.937827    2566 scope.go:117] "RemoveContainer" containerID="1ad699a0170b9eeda37c8af4ae3c9021e656c732667ca9ae16336d0e402a6a30"
Sep 18 13:36:21.948747 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:21.948727    2566 scope.go:117] "RemoveContainer" containerID="1ad699a0170b9eeda37c8af4ae3c9021e656c732667ca9ae16336d0e402a6a30"
Sep 18 13:36:21.949032 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:36:21.949008    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ad699a0170b9eeda37c8af4ae3c9021e656c732667ca9ae16336d0e402a6a30\": container with ID starting with 1ad699a0170b9eeda37c8af4ae3c9021e656c732667ca9ae16336d0e402a6a30 not found: ID does not exist" containerID="1ad699a0170b9eeda37c8af4ae3c9021e656c732667ca9ae16336d0e402a6a30"
Sep 18 13:36:21.949104 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:21.949040    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ad699a0170b9eeda37c8af4ae3c9021e656c732667ca9ae16336d0e402a6a30"} err="failed to get container status \"1ad699a0170b9eeda37c8af4ae3c9021e656c732667ca9ae16336d0e402a6a30\": rpc error: code = NotFound desc = could not find container \"1ad699a0170b9eeda37c8af4ae3c9021e656c732667ca9ae16336d0e402a6a30\": container with ID starting with 1ad699a0170b9eeda37c8af4ae3c9021e656c732667ca9ae16336d0e402a6a30 not found: ID does not exist"
Sep 18 13:36:21.970450 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:21.970424    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cab385d-33de-463b-8886-ab159b73701b-model-cache\") pod \"6cab385d-33de-463b-8886-ab159b73701b\" (UID: \"6cab385d-33de-463b-8886-ab159b73701b\") "
Sep 18 13:36:21.970566 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:21.970508    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pppq\" (UniqueName: \"kubernetes.io/projected/6cab385d-33de-463b-8886-ab159b73701b-kube-api-access-8pppq\") pod \"6cab385d-33de-463b-8886-ab159b73701b\" (UID: \"6cab385d-33de-463b-8886-ab159b73701b\") "
Sep 18 13:36:21.970566 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:21.970535    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cab385d-33de-463b-8886-ab159b73701b-dshm\") pod \"6cab385d-33de-463b-8886-ab159b73701b\" (UID: \"6cab385d-33de-463b-8886-ab159b73701b\") "
Sep 18 13:36:21.970566 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:21.970560    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cab385d-33de-463b-8886-ab159b73701b-tls-certs\") pod \"6cab385d-33de-463b-8886-ab159b73701b\" (UID: \"6cab385d-33de-463b-8886-ab159b73701b\") "
Sep 18 13:36:21.970765 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:21.970588    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cab385d-33de-463b-8886-ab159b73701b-tmp-dir\") pod \"6cab385d-33de-463b-8886-ab159b73701b\" (UID: \"6cab385d-33de-463b-8886-ab159b73701b\") "
Sep 18 13:36:21.970765 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:21.970735    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cab385d-33de-463b-8886-ab159b73701b-model-cache" (OuterVolumeSpecName: "model-cache") pod "6cab385d-33de-463b-8886-ab159b73701b" (UID: "6cab385d-33de-463b-8886-ab159b73701b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:21.970874 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:21.970746    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cab385d-33de-463b-8886-ab159b73701b-home\") pod \"6cab385d-33de-463b-8886-ab159b73701b\" (UID: \"6cab385d-33de-463b-8886-ab159b73701b\") "
Sep 18 13:36:21.970941 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:21.970919    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cab385d-33de-463b-8886-ab159b73701b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6cab385d-33de-463b-8886-ab159b73701b" (UID: "6cab385d-33de-463b-8886-ab159b73701b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:21.970941 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:21.970933    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cab385d-33de-463b-8886-ab159b73701b-home" (OuterVolumeSpecName: "home") pod "6cab385d-33de-463b-8886-ab159b73701b" (UID: "6cab385d-33de-463b-8886-ab159b73701b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:21.971276 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:21.971250    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cab385d-33de-463b-8886-ab159b73701b-tmp-dir\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:21.971276 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:21.971279    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cab385d-33de-463b-8886-ab159b73701b-home\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:21.971459 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:21.971288    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cab385d-33de-463b-8886-ab159b73701b-model-cache\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:21.972884 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:21.972865    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cab385d-33de-463b-8886-ab159b73701b-dshm" (OuterVolumeSpecName: "dshm") pod "6cab385d-33de-463b-8886-ab159b73701b" (UID: "6cab385d-33de-463b-8886-ab159b73701b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:21.972968 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:21.972930    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cab385d-33de-463b-8886-ab159b73701b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6cab385d-33de-463b-8886-ab159b73701b" (UID: "6cab385d-33de-463b-8886-ab159b73701b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:36:21.973031 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:21.973011    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cab385d-33de-463b-8886-ab159b73701b-kube-api-access-8pppq" (OuterVolumeSpecName: "kube-api-access-8pppq") pod "6cab385d-33de-463b-8886-ab159b73701b" (UID: "6cab385d-33de-463b-8886-ab159b73701b"). InnerVolumeSpecName "kube-api-access-8pppq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:36:22.071942 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:22.071908    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pppq\" (UniqueName: \"kubernetes.io/projected/6cab385d-33de-463b-8886-ab159b73701b-kube-api-access-8pppq\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:22.071942 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:22.071935    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cab385d-33de-463b-8886-ab159b73701b-dshm\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:22.071942 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:22.071947    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cab385d-33de-463b-8886-ab159b73701b-tls-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:22.085006 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:22.084963    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65ed58ca-08b3-4da7-b533-0d184bd998fe" path="/var/lib/kubelet/pods/65ed58ca-08b3-4da7-b533-0d184bd998fe/volumes"
Sep 18 13:36:22.266123 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:22.266025    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"]
Sep 18 13:36:22.276715 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:22.276681    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-021eb8/test-model-021eb8-kserve-84f4f4c465-67gtc"]
Sep 18 13:36:24.077150 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.077118    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cab385d-33de-463b-8886-ab159b73701b" path="/var/lib/kubelet/pods/6cab385d-33de-463b-8886-ab159b73701b/volumes"
Sep 18 13:36:24.321251 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.321215    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"]
Sep 18 13:36:24.321661 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.321648    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cab385d-33de-463b-8886-ab159b73701b" containerName="main"
Sep 18 13:36:24.321661 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.321662    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cab385d-33de-463b-8886-ab159b73701b" containerName="main"
Sep 18 13:36:24.321757 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.321671    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65ed58ca-08b3-4da7-b533-0d184bd998fe" containerName="main"
Sep 18 13:36:24.321757 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.321677    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="65ed58ca-08b3-4da7-b533-0d184bd998fe" containerName="main"
Sep 18 13:36:24.321757 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.321754    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cab385d-33de-463b-8886-ab159b73701b" containerName="main"
Sep 18 13:36:24.321854 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.321763    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="65ed58ca-08b3-4da7-b533-0d184bd998fe" containerName="main"
Sep 18 13:36:24.325976 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.325956    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:36:24.330120 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.330049    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5be8a3\"/\"e2e-conflict-b-158c7039-kserve-self-signed-certs\""
Sep 18 13:36:24.396431 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.396396    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"]
Sep 18 13:36:24.396866 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.396843    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/296a73a3-b94f-44d7-949c-a38b18c8f66d-model-cache\") pod \"e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf\" (UID: \"296a73a3-b94f-44d7-949c-a38b18c8f66d\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:36:24.396913 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.396885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/296a73a3-b94f-44d7-949c-a38b18c8f66d-dshm\") pod \"e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf\" (UID: \"296a73a3-b94f-44d7-949c-a38b18c8f66d\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:36:24.396969 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.396946    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbpcp\" (UniqueName: \"kubernetes.io/projected/296a73a3-b94f-44d7-949c-a38b18c8f66d-kube-api-access-jbpcp\") pod \"e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf\" (UID: \"296a73a3-b94f-44d7-949c-a38b18c8f66d\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:36:24.397020 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.396974    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/296a73a3-b94f-44d7-949c-a38b18c8f66d-home\") pod \"e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf\" (UID: \"296a73a3-b94f-44d7-949c-a38b18c8f66d\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:36:24.397058 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.397011    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/296a73a3-b94f-44d7-949c-a38b18c8f66d-tls-certs\") pod \"e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf\" (UID: \"296a73a3-b94f-44d7-949c-a38b18c8f66d\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:36:24.397138 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.397124    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/296a73a3-b94f-44d7-949c-a38b18c8f66d-tmp-dir\") pod \"e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf\" (UID: \"296a73a3-b94f-44d7-949c-a38b18c8f66d\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:36:24.498212 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.498179    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbpcp\" (UniqueName: \"kubernetes.io/projected/296a73a3-b94f-44d7-949c-a38b18c8f66d-kube-api-access-jbpcp\") pod \"e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf\" (UID: \"296a73a3-b94f-44d7-949c-a38b18c8f66d\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:36:24.498412 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.498223    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/296a73a3-b94f-44d7-949c-a38b18c8f66d-home\") pod \"e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf\" (UID: \"296a73a3-b94f-44d7-949c-a38b18c8f66d\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:36:24.498412 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.498240    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/296a73a3-b94f-44d7-949c-a38b18c8f66d-tls-certs\") pod \"e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf\" (UID: \"296a73a3-b94f-44d7-949c-a38b18c8f66d\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:36:24.498412 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.498294    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/296a73a3-b94f-44d7-949c-a38b18c8f66d-tmp-dir\") pod \"e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf\" (UID: \"296a73a3-b94f-44d7-949c-a38b18c8f66d\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:36:24.498412 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.498358    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/296a73a3-b94f-44d7-949c-a38b18c8f66d-model-cache\") pod \"e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf\" (UID: \"296a73a3-b94f-44d7-949c-a38b18c8f66d\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:36:24.498412 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.498396    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/296a73a3-b94f-44d7-949c-a38b18c8f66d-dshm\") pod \"e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf\" (UID: \"296a73a3-b94f-44d7-949c-a38b18c8f66d\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:36:24.498738 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.498672    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/296a73a3-b94f-44d7-949c-a38b18c8f66d-home\") pod \"e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf\" (UID: \"296a73a3-b94f-44d7-949c-a38b18c8f66d\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:36:24.498738 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.498705    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/296a73a3-b94f-44d7-949c-a38b18c8f66d-tmp-dir\") pod \"e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf\" (UID: \"296a73a3-b94f-44d7-949c-a38b18c8f66d\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:36:24.498806 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.498739    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/296a73a3-b94f-44d7-949c-a38b18c8f66d-model-cache\") pod \"e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf\" (UID: \"296a73a3-b94f-44d7-949c-a38b18c8f66d\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:36:24.500569 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.500549    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/296a73a3-b94f-44d7-949c-a38b18c8f66d-dshm\") pod \"e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf\" (UID: \"296a73a3-b94f-44d7-949c-a38b18c8f66d\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:36:24.500857 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.500840    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/296a73a3-b94f-44d7-949c-a38b18c8f66d-tls-certs\") pod \"e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf\" (UID: \"296a73a3-b94f-44d7-949c-a38b18c8f66d\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:36:24.516817 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.516789    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbpcp\" (UniqueName: \"kubernetes.io/projected/296a73a3-b94f-44d7-949c-a38b18c8f66d-kube-api-access-jbpcp\") pod \"e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf\" (UID: \"296a73a3-b94f-44d7-949c-a38b18c8f66d\") " pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:36:24.636042 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.635999    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:36:24.956590 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:24.956410    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"]
Sep 18 13:36:25.970681 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:25.970644    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf" event={"ID":"296a73a3-b94f-44d7-949c-a38b18c8f66d","Type":"ContainerStarted","Data":"7af492cd718602ae6555dc65640a463db2582622d28f42a4d555bcbfe40cfa14"}
Sep 18 13:36:25.971089 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:25.970693    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf" event={"ID":"296a73a3-b94f-44d7-949c-a38b18c8f66d","Type":"ContainerStarted","Data":"6f6e9bae21743cf497e8415756732bd2f9902a2f9d5699dbd0823e55e12419df"}
Sep 18 13:36:26.007161 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:26.007107    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf" podStartSLOduration=2.007089916 podStartE2EDuration="2.007089916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:36:26.00604683 +0000 UTC m=+1294.440923235" watchObservedRunningTime="2026-09-18 13:36:26.007089916 +0000 UTC m=+1294.441966347"
Sep 18 13:36:27.913756 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:27.913722    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc"
Sep 18 13:36:29.526145 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:29.526108    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"]
Sep 18 13:36:29.526612 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:29.526325    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt" podUID="03f2a203-ca49-4f5c-bcdd-58fc57490d94" containerName="istio-proxy" containerID="cri-o://f684fe0c7916f50534650b5eb38ed5a70cf469f8150b2635eb91f439b981e87b" gracePeriod=30
Sep 18 13:36:34.009879 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.009835    2566 patch_prober.go:28] interesting pod/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 13:36:34.010348 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.009897    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt" podUID="03f2a203-ca49-4f5c-bcdd-58fc57490d94" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 13:36:34.637104 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.637053    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:36:34.637251 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.637136    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:36:34.678997 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.678810    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:36:34.824911 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.824884    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:36:34.913367 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.913334    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istiod-ca-cert\") pod \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") "
Sep 18 13:36:34.913567 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.913410    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-credential-socket\") pod \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") "
Sep 18 13:36:34.913567 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.913441    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm66l\" (UniqueName: \"kubernetes.io/projected/03f2a203-ca49-4f5c-bcdd-58fc57490d94-kube-api-access-pm66l\") pod \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") "
Sep 18 13:36:34.913567 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.913464    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istio-data\") pod \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") "
Sep 18 13:36:34.913567 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.913510    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istio-token\") pod \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") "
Sep 18 13:36:34.913567 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.913551    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istio-podinfo\") pod \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") "
Sep 18 13:36:34.913872 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.913823    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istio-data" (OuterVolumeSpecName: "istio-data") pod "03f2a203-ca49-4f5c-bcdd-58fc57490d94" (UID: "03f2a203-ca49-4f5c-bcdd-58fc57490d94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:34.913918 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.913865    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "03f2a203-ca49-4f5c-bcdd-58fc57490d94" (UID: "03f2a203-ca49-4f5c-bcdd-58fc57490d94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:36:34.914026 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.914001    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-workload-certs\") pod \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") "
Sep 18 13:36:34.914282 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.914255    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "03f2a203-ca49-4f5c-bcdd-58fc57490d94" (UID: "03f2a203-ca49-4f5c-bcdd-58fc57490d94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:34.914364 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.914311    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "03f2a203-ca49-4f5c-bcdd-58fc57490d94" (UID: "03f2a203-ca49-4f5c-bcdd-58fc57490d94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:34.914481 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.914457    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istio-envoy\") pod \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") "
Sep 18 13:36:34.914621 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.914511    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-workload-socket\") pod \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\" (UID: \"03f2a203-ca49-4f5c-bcdd-58fc57490d94\") "
Sep 18 13:36:34.915007 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.914979    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-workload-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:34.915007 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.914997    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "03f2a203-ca49-4f5c-bcdd-58fc57490d94" (UID: "03f2a203-ca49-4f5c-bcdd-58fc57490d94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:34.915165 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.915007    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istiod-ca-cert\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:34.915165 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.915040    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-credential-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:34.915165 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.915054    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istio-data\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:34.916091 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.916046    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f2a203-ca49-4f5c-bcdd-58fc57490d94-kube-api-access-pm66l" (OuterVolumeSpecName: "kube-api-access-pm66l") pod "03f2a203-ca49-4f5c-bcdd-58fc57490d94" (UID: "03f2a203-ca49-4f5c-bcdd-58fc57490d94"). InnerVolumeSpecName "kube-api-access-pm66l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:36:34.916247 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.916221    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istio-token" (OuterVolumeSpecName: "istio-token") pod "03f2a203-ca49-4f5c-bcdd-58fc57490d94" (UID: "03f2a203-ca49-4f5c-bcdd-58fc57490d94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:36:34.916412 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.916383    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "03f2a203-ca49-4f5c-bcdd-58fc57490d94" (UID: "03f2a203-ca49-4f5c-bcdd-58fc57490d94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 13:36:34.916908 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:34.916880    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "03f2a203-ca49-4f5c-bcdd-58fc57490d94" (UID: "03f2a203-ca49-4f5c-bcdd-58fc57490d94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:35.015948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:35.015912    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istio-token\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:35.015948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:35.015945    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istio-podinfo\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:35.015948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:35.015954    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-istio-envoy\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:35.016387 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:35.015963    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f2a203-ca49-4f5c-bcdd-58fc57490d94-workload-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:35.016387 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:35.015972    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm66l\" (UniqueName: \"kubernetes.io/projected/03f2a203-ca49-4f5c-bcdd-58fc57490d94-kube-api-access-pm66l\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:35.026773 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:35.026742    2566 generic.go:356] "Generic (PLEG): container finished" podID="03f2a203-ca49-4f5c-bcdd-58fc57490d94" containerID="f684fe0c7916f50534650b5eb38ed5a70cf469f8150b2635eb91f439b981e87b" exitCode=0
Sep 18 13:36:35.026933 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:35.026801    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"
Sep 18 13:36:35.026933 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:35.026833    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt" event={"ID":"03f2a203-ca49-4f5c-bcdd-58fc57490d94","Type":"ContainerDied","Data":"f684fe0c7916f50534650b5eb38ed5a70cf469f8150b2635eb91f439b981e87b"}
Sep 18 13:36:35.026933 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:35.026892    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt" event={"ID":"03f2a203-ca49-4f5c-bcdd-58fc57490d94","Type":"ContainerDied","Data":"61aafcf04b474f3fbb120d57eea4605d5d8672bcc7fd60c5e2afd0fa1a48506c"}
Sep 18 13:36:35.026933 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:35.026923    2566 scope.go:117] "RemoveContainer" containerID="f684fe0c7916f50534650b5eb38ed5a70cf469f8150b2635eb91f439b981e87b"
Sep 18 13:36:35.037730 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:35.037650    2566 scope.go:117] "RemoveContainer" containerID="fb4818768db14ee04b5bc6d801cd5345a07c92a288fec149cf04f7f47c7e090d"
Sep 18 13:36:35.072642 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:35.072428    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:36:35.082503 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:35.082468    2566 scope.go:117] "RemoveContainer" containerID="f684fe0c7916f50534650b5eb38ed5a70cf469f8150b2635eb91f439b981e87b"
Sep 18 13:36:35.083001 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:36:35.082959    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f684fe0c7916f50534650b5eb38ed5a70cf469f8150b2635eb91f439b981e87b\": container with ID starting with f684fe0c7916f50534650b5eb38ed5a70cf469f8150b2635eb91f439b981e87b not found: ID does not exist" containerID="f684fe0c7916f50534650b5eb38ed5a70cf469f8150b2635eb91f439b981e87b"
Sep 18 13:36:35.083139 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:35.083000    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f684fe0c7916f50534650b5eb38ed5a70cf469f8150b2635eb91f439b981e87b"} err="failed to get container status \"f684fe0c7916f50534650b5eb38ed5a70cf469f8150b2635eb91f439b981e87b\": rpc error: code = NotFound desc = could not find container \"f684fe0c7916f50534650b5eb38ed5a70cf469f8150b2635eb91f439b981e87b\": container with ID starting with f684fe0c7916f50534650b5eb38ed5a70cf469f8150b2635eb91f439b981e87b not found: ID does not exist"
Sep 18 13:36:35.083139 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:35.083028    2566 scope.go:117] "RemoveContainer" containerID="fb4818768db14ee04b5bc6d801cd5345a07c92a288fec149cf04f7f47c7e090d"
Sep 18 13:36:35.083399 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:36:35.083363    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4818768db14ee04b5bc6d801cd5345a07c92a288fec149cf04f7f47c7e090d\": container with ID starting with fb4818768db14ee04b5bc6d801cd5345a07c92a288fec149cf04f7f47c7e090d not found: ID does not exist" containerID="fb4818768db14ee04b5bc6d801cd5345a07c92a288fec149cf04f7f47c7e090d"
Sep 18 13:36:35.083484 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:35.083398    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4818768db14ee04b5bc6d801cd5345a07c92a288fec149cf04f7f47c7e090d"} err="failed to get container status \"fb4818768db14ee04b5bc6d801cd5345a07c92a288fec149cf04f7f47c7e090d\": rpc error: code = NotFound desc = could not find container \"fb4818768db14ee04b5bc6d801cd5345a07c92a288fec149cf04f7f47c7e090d\": container with ID starting with fb4818768db14ee04b5bc6d801cd5345a07c92a288fec149cf04f7f47c7e090d not found: ID does not exist"
Sep 18 13:36:35.084271 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:35.084249    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"]
Sep 18 13:36:35.125137 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:35.125103    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-34aeca-openshift-default-74bc449fb4-9tbzt"]
Sep 18 13:36:36.084039 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:36.084005    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03f2a203-ca49-4f5c-bcdd-58fc57490d94" path="/var/lib/kubelet/pods/03f2a203-ca49-4f5c-bcdd-58fc57490d94/volumes"
Sep 18 13:36:40.465341 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:40.465302    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"]
Sep 18 13:36:44.092379 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:44.092346    2566 generic.go:356] "Generic (PLEG): container finished" podID="3e87c7b2-c903-4be2-9039-3499f78e2cbb" containerID="f1659a0bde9830086be0299d3bc16d4c61a28de557c91418319db5a7a3bcd590" exitCode=0
Sep 18 13:36:44.092955 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:44.092426    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j" event={"ID":"3e87c7b2-c903-4be2-9039-3499f78e2cbb","Type":"ContainerDied","Data":"f1659a0bde9830086be0299d3bc16d4c61a28de557c91418319db5a7a3bcd590"}
Sep 18 13:36:44.093053 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:44.093035    2566 scope.go:117] "RemoveContainer" containerID="f1659a0bde9830086be0299d3bc16d4c61a28de557c91418319db5a7a3bcd590"
Sep 18 13:36:44.094272 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:44.094242    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:36:44.094383 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:44.094328    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:36:44.094445 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:44.094399    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:36:45.100243 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:45.100209    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j" event={"ID":"3e87c7b2-c903-4be2-9039-3499f78e2cbb","Type":"ContainerStarted","Data":"9a30be30d80af2fced517fcdd791cad6e976b0ff51c7d86fd00fed961e336ace"}
Sep 18 13:36:45.234273 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:45.234232    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:45.234466 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:45.234444    2566 patch_prober.go:28] interesting pod/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 18 13:36:45.234520 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:45.234491    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j" podUID="3e87c7b2-c903-4be2-9039-3499f78e2cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 18 13:36:46.235421 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:46.235382    2566 patch_prober.go:28] interesting pod/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 18 13:36:46.235836 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:46.235451    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j" podUID="3e87c7b2-c903-4be2-9039-3499f78e2cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 18 13:36:47.234621 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:47.234552    2566 patch_prober.go:28] interesting pod/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 18 13:36:47.234813 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:47.234672    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j" podUID="3e87c7b2-c903-4be2-9039-3499f78e2cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 18 13:36:48.235258 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:48.235213    2566 patch_prober.go:28] interesting pod/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 18 13:36:48.235702 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:48.235274    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j" podUID="3e87c7b2-c903-4be2-9039-3499f78e2cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 18 13:36:49.235057 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:49.235022    2566 patch_prober.go:28] interesting pod/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Sep 18 13:36:49.235241 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:49.235080    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j" podUID="3e87c7b2-c903-4be2-9039-3499f78e2cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Sep 18 13:36:49.263839 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:49.263774    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"]
Sep 18 13:36:49.264350 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:49.264178    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb" podUID="03837341-329a-4b27-827e-4b95a242d96e" containerName="istio-proxy" containerID="cri-o://f0d807229cdc7242e336f3899ce9a9b65903217702c38d5983ea307b9b991fb5" gracePeriod=30
Sep 18 13:36:50.239337 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:50.239298    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:50.239870 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:50.239851    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:50.240635 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:50.240614    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:36:52.175430 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:52.175398    2566 scope.go:117] "RemoveContainer" containerID="23c37de3b316add730d3ff03e7b3311d58606f4dd28377bee71d9ed1e487ca00"
Sep 18 13:36:52.184247 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:52.184221    2566 scope.go:117] "RemoveContainer" containerID="19ceabdc1eae44ad206b57f859affea7c8dc374b89a4c8da5befb8d414bb7b00"
Sep 18 13:36:52.192654 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:52.192634    2566 scope.go:117] "RemoveContainer" containerID="0f71a65fae577f2b4469d84e41bcf887bfa1dfb2be0f3d07ff80c913391d6c73"
Sep 18 13:36:54.535677 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.535576    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:36:54.650890 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.650857    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03837341-329a-4b27-827e-4b95a242d96e-istio-podinfo\") pod \"03837341-329a-4b27-827e-4b95a242d96e\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") "
Sep 18 13:36:54.651081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.650916    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03837341-329a-4b27-827e-4b95a242d96e-istio-token\") pod \"03837341-329a-4b27-827e-4b95a242d96e\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") "
Sep 18 13:36:54.651081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.650940    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03837341-329a-4b27-827e-4b95a242d96e-istiod-ca-cert\") pod \"03837341-329a-4b27-827e-4b95a242d96e\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") "
Sep 18 13:36:54.651081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.650967    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-istio-data\") pod \"03837341-329a-4b27-827e-4b95a242d96e\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") "
Sep 18 13:36:54.651081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.650994    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-credential-socket\") pod \"03837341-329a-4b27-827e-4b95a242d96e\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") "
Sep 18 13:36:54.651081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.651028    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-workload-certs\") pod \"03837341-329a-4b27-827e-4b95a242d96e\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") "
Sep 18 13:36:54.651081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.651066    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97lwc\" (UniqueName: \"kubernetes.io/projected/03837341-329a-4b27-827e-4b95a242d96e-kube-api-access-97lwc\") pod \"03837341-329a-4b27-827e-4b95a242d96e\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") "
Sep 18 13:36:54.651398 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.651138    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-workload-socket\") pod \"03837341-329a-4b27-827e-4b95a242d96e\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") "
Sep 18 13:36:54.651398 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.651165    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-istio-envoy\") pod \"03837341-329a-4b27-827e-4b95a242d96e\" (UID: \"03837341-329a-4b27-827e-4b95a242d96e\") "
Sep 18 13:36:54.651398 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.651301    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "03837341-329a-4b27-827e-4b95a242d96e" (UID: "03837341-329a-4b27-827e-4b95a242d96e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:54.651547 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.651314    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-istio-data" (OuterVolumeSpecName: "istio-data") pod "03837341-329a-4b27-827e-4b95a242d96e" (UID: "03837341-329a-4b27-827e-4b95a242d96e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:54.651547 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.651401    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "03837341-329a-4b27-827e-4b95a242d96e" (UID: "03837341-329a-4b27-827e-4b95a242d96e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:54.651547 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.651464    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "03837341-329a-4b27-827e-4b95a242d96e" (UID: "03837341-329a-4b27-827e-4b95a242d96e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:54.651734 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.651659    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03837341-329a-4b27-827e-4b95a242d96e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "03837341-329a-4b27-827e-4b95a242d96e" (UID: "03837341-329a-4b27-827e-4b95a242d96e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:36:54.651734 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.651683    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-istio-data\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:54.651734 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.651704    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-credential-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:54.651734 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.651714    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-workload-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:54.651734 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.651723    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-workload-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:54.653252 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.653220    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03837341-329a-4b27-827e-4b95a242d96e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "03837341-329a-4b27-827e-4b95a242d96e" (UID: "03837341-329a-4b27-827e-4b95a242d96e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 13:36:54.653386 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.653318    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03837341-329a-4b27-827e-4b95a242d96e-istio-token" (OuterVolumeSpecName: "istio-token") pod "03837341-329a-4b27-827e-4b95a242d96e" (UID: "03837341-329a-4b27-827e-4b95a242d96e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:36:54.653386 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.653373    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "03837341-329a-4b27-827e-4b95a242d96e" (UID: "03837341-329a-4b27-827e-4b95a242d96e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:54.653485 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.653409    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03837341-329a-4b27-827e-4b95a242d96e-kube-api-access-97lwc" (OuterVolumeSpecName: "kube-api-access-97lwc") pod "03837341-329a-4b27-827e-4b95a242d96e" (UID: "03837341-329a-4b27-827e-4b95a242d96e"). InnerVolumeSpecName "kube-api-access-97lwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:36:54.752750 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.752714    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03837341-329a-4b27-827e-4b95a242d96e-istio-envoy\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:54.752750 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.752746    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03837341-329a-4b27-827e-4b95a242d96e-istio-podinfo\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:54.752750 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.752757    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03837341-329a-4b27-827e-4b95a242d96e-istio-token\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:54.752996 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.752768    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03837341-329a-4b27-827e-4b95a242d96e-istiod-ca-cert\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:54.752996 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:54.752776    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97lwc\" (UniqueName: \"kubernetes.io/projected/03837341-329a-4b27-827e-4b95a242d96e-kube-api-access-97lwc\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:55.163312 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.163278    2566 generic.go:356] "Generic (PLEG): container finished" podID="03837341-329a-4b27-827e-4b95a242d96e" containerID="f0d807229cdc7242e336f3899ce9a9b65903217702c38d5983ea307b9b991fb5" exitCode=0
Sep 18 13:36:55.163527 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.163326    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"
Sep 18 13:36:55.163527 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.163365    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb" event={"ID":"03837341-329a-4b27-827e-4b95a242d96e","Type":"ContainerDied","Data":"f0d807229cdc7242e336f3899ce9a9b65903217702c38d5983ea307b9b991fb5"}
Sep 18 13:36:55.163527 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.163428    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb" event={"ID":"03837341-329a-4b27-827e-4b95a242d96e","Type":"ContainerDied","Data":"5637289a147081c450c0333a9f47aec6afd6dd4e468501d77082137a6ca36a9f"}
Sep 18 13:36:55.163527 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.163458    2566 scope.go:117] "RemoveContainer" containerID="f0d807229cdc7242e336f3899ce9a9b65903217702c38d5983ea307b9b991fb5"
Sep 18 13:36:55.174049 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.174028    2566 scope.go:117] "RemoveContainer" containerID="f3610132459dba62fbe9c245d6b469865045dc2edaf1042d83e49417b3424ac7"
Sep 18 13:36:55.182610 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.182561    2566 scope.go:117] "RemoveContainer" containerID="f0d807229cdc7242e336f3899ce9a9b65903217702c38d5983ea307b9b991fb5"
Sep 18 13:36:55.182951 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:36:55.182922    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0d807229cdc7242e336f3899ce9a9b65903217702c38d5983ea307b9b991fb5\": container with ID starting with f0d807229cdc7242e336f3899ce9a9b65903217702c38d5983ea307b9b991fb5 not found: ID does not exist" containerID="f0d807229cdc7242e336f3899ce9a9b65903217702c38d5983ea307b9b991fb5"
Sep 18 13:36:55.183049 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.182960    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d807229cdc7242e336f3899ce9a9b65903217702c38d5983ea307b9b991fb5"} err="failed to get container status \"f0d807229cdc7242e336f3899ce9a9b65903217702c38d5983ea307b9b991fb5\": rpc error: code = NotFound desc = could not find container \"f0d807229cdc7242e336f3899ce9a9b65903217702c38d5983ea307b9b991fb5\": container with ID starting with f0d807229cdc7242e336f3899ce9a9b65903217702c38d5983ea307b9b991fb5 not found: ID does not exist"
Sep 18 13:36:55.183049 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.182986    2566 scope.go:117] "RemoveContainer" containerID="f3610132459dba62fbe9c245d6b469865045dc2edaf1042d83e49417b3424ac7"
Sep 18 13:36:55.183304 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:36:55.183284    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3610132459dba62fbe9c245d6b469865045dc2edaf1042d83e49417b3424ac7\": container with ID starting with f3610132459dba62fbe9c245d6b469865045dc2edaf1042d83e49417b3424ac7 not found: ID does not exist" containerID="f3610132459dba62fbe9c245d6b469865045dc2edaf1042d83e49417b3424ac7"
Sep 18 13:36:55.183361 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.183310    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3610132459dba62fbe9c245d6b469865045dc2edaf1042d83e49417b3424ac7"} err="failed to get container status \"f3610132459dba62fbe9c245d6b469865045dc2edaf1042d83e49417b3424ac7\": rpc error: code = NotFound desc = could not find container \"f3610132459dba62fbe9c245d6b469865045dc2edaf1042d83e49417b3424ac7\": container with ID starting with f3610132459dba62fbe9c245d6b469865045dc2edaf1042d83e49417b3424ac7 not found: ID does not exist"
Sep 18 13:36:55.194723 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.194697    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"]
Sep 18 13:36:55.198026 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.198001    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-021eb8-openshift-default-5965d6499b-ztkhb"]
Sep 18 13:36:55.485231 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.485130    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw" podUID="1d3d0335-76e3-4912-9def-f50f888bc0cb" containerName="main" containerID="cri-o://89c930af069401315cfe60927385d752d29b8aa7e19eac12c29d72abd65fa0fc" gracePeriod=45
Sep 18 13:36:55.737336 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.737273    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:55.763024 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.762991    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-692hq\" (UniqueName: \"kubernetes.io/projected/1d3d0335-76e3-4912-9def-f50f888bc0cb-kube-api-access-692hq\") pod \"1d3d0335-76e3-4912-9def-f50f888bc0cb\" (UID: \"1d3d0335-76e3-4912-9def-f50f888bc0cb\") "
Sep 18 13:36:55.763189 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.763041    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d3d0335-76e3-4912-9def-f50f888bc0cb-dshm\") pod \"1d3d0335-76e3-4912-9def-f50f888bc0cb\" (UID: \"1d3d0335-76e3-4912-9def-f50f888bc0cb\") "
Sep 18 13:36:55.763189 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.763091    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3d0335-76e3-4912-9def-f50f888bc0cb-tmp-dir\") pod \"1d3d0335-76e3-4912-9def-f50f888bc0cb\" (UID: \"1d3d0335-76e3-4912-9def-f50f888bc0cb\") "
Sep 18 13:36:55.763333 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.763180    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d3d0335-76e3-4912-9def-f50f888bc0cb-home\") pod \"1d3d0335-76e3-4912-9def-f50f888bc0cb\" (UID: \"1d3d0335-76e3-4912-9def-f50f888bc0cb\") "
Sep 18 13:36:55.763333 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.763228    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d3d0335-76e3-4912-9def-f50f888bc0cb-tls-certs\") pod \"1d3d0335-76e3-4912-9def-f50f888bc0cb\" (UID: \"1d3d0335-76e3-4912-9def-f50f888bc0cb\") "
Sep 18 13:36:55.763333 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.763257    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d3d0335-76e3-4912-9def-f50f888bc0cb-model-cache\") pod \"1d3d0335-76e3-4912-9def-f50f888bc0cb\" (UID: \"1d3d0335-76e3-4912-9def-f50f888bc0cb\") "
Sep 18 13:36:55.763551 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.763345    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3d0335-76e3-4912-9def-f50f888bc0cb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d3d0335-76e3-4912-9def-f50f888bc0cb" (UID: "1d3d0335-76e3-4912-9def-f50f888bc0cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:55.763551 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.763430    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3d0335-76e3-4912-9def-f50f888bc0cb-home" (OuterVolumeSpecName: "home") pod "1d3d0335-76e3-4912-9def-f50f888bc0cb" (UID: "1d3d0335-76e3-4912-9def-f50f888bc0cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:55.764848 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.763728    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3d0335-76e3-4912-9def-f50f888bc0cb-tmp-dir\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:55.764848 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.763755    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d3d0335-76e3-4912-9def-f50f888bc0cb-home\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:55.764848 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.763893    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3d0335-76e3-4912-9def-f50f888bc0cb-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d3d0335-76e3-4912-9def-f50f888bc0cb" (UID: "1d3d0335-76e3-4912-9def-f50f888bc0cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:55.766029 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.765972    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d3d0335-76e3-4912-9def-f50f888bc0cb-kube-api-access-692hq" (OuterVolumeSpecName: "kube-api-access-692hq") pod "1d3d0335-76e3-4912-9def-f50f888bc0cb" (UID: "1d3d0335-76e3-4912-9def-f50f888bc0cb"). InnerVolumeSpecName "kube-api-access-692hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:36:55.766272 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.766240    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d3d0335-76e3-4912-9def-f50f888bc0cb-dshm" (OuterVolumeSpecName: "dshm") pod "1d3d0335-76e3-4912-9def-f50f888bc0cb" (UID: "1d3d0335-76e3-4912-9def-f50f888bc0cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:55.766538 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.766515    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d3d0335-76e3-4912-9def-f50f888bc0cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d3d0335-76e3-4912-9def-f50f888bc0cb" (UID: "1d3d0335-76e3-4912-9def-f50f888bc0cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:36:55.864399 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.864369    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-692hq\" (UniqueName: \"kubernetes.io/projected/1d3d0335-76e3-4912-9def-f50f888bc0cb-kube-api-access-692hq\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:55.864399 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.864396    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d3d0335-76e3-4912-9def-f50f888bc0cb-dshm\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:55.864399 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.864406    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d3d0335-76e3-4912-9def-f50f888bc0cb-tls-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:55.864655 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:55.864425    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d3d0335-76e3-4912-9def-f50f888bc0cb-model-cache\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:36:56.076990 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:56.076909    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03837341-329a-4b27-827e-4b95a242d96e" path="/var/lib/kubelet/pods/03837341-329a-4b27-827e-4b95a242d96e/volumes"
Sep 18 13:36:56.170426 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:56.170387    2566 generic.go:356] "Generic (PLEG): container finished" podID="1d3d0335-76e3-4912-9def-f50f888bc0cb" containerID="89c930af069401315cfe60927385d752d29b8aa7e19eac12c29d72abd65fa0fc" exitCode=0
Sep 18 13:36:56.170632 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:56.170434    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"
Sep 18 13:36:56.170632 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:56.170476    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw" event={"ID":"1d3d0335-76e3-4912-9def-f50f888bc0cb","Type":"ContainerDied","Data":"89c930af069401315cfe60927385d752d29b8aa7e19eac12c29d72abd65fa0fc"}
Sep 18 13:36:56.170632 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:56.170522    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw" event={"ID":"1d3d0335-76e3-4912-9def-f50f888bc0cb","Type":"ContainerDied","Data":"4f0b5d02baa62b63ffecfeb608b35f239ab87a570b2440f79b742afca5bd3b3e"}
Sep 18 13:36:56.170632 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:56.170545    2566 scope.go:117] "RemoveContainer" containerID="89c930af069401315cfe60927385d752d29b8aa7e19eac12c29d72abd65fa0fc"
Sep 18 13:36:56.179412 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:56.179393    2566 scope.go:117] "RemoveContainer" containerID="89c930af069401315cfe60927385d752d29b8aa7e19eac12c29d72abd65fa0fc"
Sep 18 13:36:56.179721 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:36:56.179701    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89c930af069401315cfe60927385d752d29b8aa7e19eac12c29d72abd65fa0fc\": container with ID starting with 89c930af069401315cfe60927385d752d29b8aa7e19eac12c29d72abd65fa0fc not found: ID does not exist" containerID="89c930af069401315cfe60927385d752d29b8aa7e19eac12c29d72abd65fa0fc"
Sep 18 13:36:56.179776 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:56.179728    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c930af069401315cfe60927385d752d29b8aa7e19eac12c29d72abd65fa0fc"} err="failed to get container status \"89c930af069401315cfe60927385d752d29b8aa7e19eac12c29d72abd65fa0fc\": rpc error: code = NotFound desc = could not find container \"89c930af069401315cfe60927385d752d29b8aa7e19eac12c29d72abd65fa0fc\": container with ID starting with 89c930af069401315cfe60927385d752d29b8aa7e19eac12c29d72abd65fa0fc not found: ID does not exist"
Sep 18 13:36:56.192264 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:56.192239    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"]
Sep 18 13:36:56.197453 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:56.197430    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-b81e47/noten-model-b81e47-kserve-86c9b8f9ff-86dxw"]
Sep 18 13:36:58.078365 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:36:58.078329    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d3d0335-76e3-4912-9def-f50f888bc0cb" path="/var/lib/kubelet/pods/1d3d0335-76e3-4912-9def-f50f888bc0cb/volumes"
Sep 18 13:37:00.336192 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.336147    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp"]
Sep 18 13:37:00.336770 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.336750    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03837341-329a-4b27-827e-4b95a242d96e" containerName="istio-proxy"
Sep 18 13:37:00.336770 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.336771    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="03837341-329a-4b27-827e-4b95a242d96e" containerName="istio-proxy"
Sep 18 13:37:00.336916 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.336792    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03f2a203-ca49-4f5c-bcdd-58fc57490d94" containerName="istio-proxy"
Sep 18 13:37:00.336916 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.336800    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="03f2a203-ca49-4f5c-bcdd-58fc57490d94" containerName="istio-proxy"
Sep 18 13:37:00.336916 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.336810    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03837341-329a-4b27-827e-4b95a242d96e" containerName="istio-proxy"
Sep 18 13:37:00.336916 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.336820    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="03837341-329a-4b27-827e-4b95a242d96e" containerName="istio-proxy"
Sep 18 13:37:00.336916 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.336832    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03f2a203-ca49-4f5c-bcdd-58fc57490d94" containerName="istio-proxy"
Sep 18 13:37:00.336916 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.336840    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="03f2a203-ca49-4f5c-bcdd-58fc57490d94" containerName="istio-proxy"
Sep 18 13:37:00.336916 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.336852    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d3d0335-76e3-4912-9def-f50f888bc0cb" containerName="main"
Sep 18 13:37:00.336916 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.336860    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d3d0335-76e3-4912-9def-f50f888bc0cb" containerName="main"
Sep 18 13:37:00.337304 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.336987    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="03f2a203-ca49-4f5c-bcdd-58fc57490d94" containerName="istio-proxy"
Sep 18 13:37:00.337304 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.337002    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="03f2a203-ca49-4f5c-bcdd-58fc57490d94" containerName="istio-proxy"
Sep 18 13:37:00.337304 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.337011    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="03837341-329a-4b27-827e-4b95a242d96e" containerName="istio-proxy"
Sep 18 13:37:00.337304 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.337021    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d3d0335-76e3-4912-9def-f50f888bc0cb" containerName="main"
Sep 18 13:37:00.337304 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.337030    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="03837341-329a-4b27-827e-4b95a242d96e" containerName="istio-proxy"
Sep 18 13:37:00.342028 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.342001    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp"
Sep 18 13:37:00.344509 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.344487    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-537abe\""
Sep 18 13:37:00.350070 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.350036    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp"]
Sep 18 13:37:00.414149 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.414115    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558b86b8-b911-4ce8-adaa-4b4af2f6833e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp\" (UID: \"558b86b8-b911-4ce8-adaa-4b4af2f6833e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp"
Sep 18 13:37:00.414337 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.414176    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-777p9\" (UniqueName: \"kubernetes.io/projected/558b86b8-b911-4ce8-adaa-4b4af2f6833e-kube-api-access-777p9\") pod \"payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp\" (UID: \"558b86b8-b911-4ce8-adaa-4b4af2f6833e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp"
Sep 18 13:37:00.441643 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.441581    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9"]
Sep 18 13:37:00.448715 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.448680    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9"
Sep 18 13:37:00.451964 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.451937    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-537abe-dockercfg-mrjwl\""
Sep 18 13:37:00.466813 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.466781    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9"]
Sep 18 13:37:00.515427 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.515397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fv75\" (UniqueName: \"kubernetes.io/projected/c69ee1a0-f4ea-4f73-a53a-e435c99524bb-kube-api-access-9fv75\") pod \"payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9\" (UID: \"c69ee1a0-f4ea-4f73-a53a-e435c99524bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9"
Sep 18 13:37:00.515676 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.515498    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558b86b8-b911-4ce8-adaa-4b4af2f6833e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp\" (UID: \"558b86b8-b911-4ce8-adaa-4b4af2f6833e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp"
Sep 18 13:37:00.515676 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.515530    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69ee1a0-f4ea-4f73-a53a-e435c99524bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9\" (UID: \"c69ee1a0-f4ea-4f73-a53a-e435c99524bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9"
Sep 18 13:37:00.515676 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.515555    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-777p9\" (UniqueName: \"kubernetes.io/projected/558b86b8-b911-4ce8-adaa-4b4af2f6833e-kube-api-access-777p9\") pod \"payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp\" (UID: \"558b86b8-b911-4ce8-adaa-4b4af2f6833e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp"
Sep 18 13:37:00.516320 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.516297    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558b86b8-b911-4ce8-adaa-4b4af2f6833e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp\" (UID: \"558b86b8-b911-4ce8-adaa-4b4af2f6833e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp"
Sep 18 13:37:00.531288 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.531261    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-777p9\" (UniqueName: \"kubernetes.io/projected/558b86b8-b911-4ce8-adaa-4b4af2f6833e-kube-api-access-777p9\") pod \"payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp\" (UID: \"558b86b8-b911-4ce8-adaa-4b4af2f6833e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp"
Sep 18 13:37:00.617133 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.617100    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fv75\" (UniqueName: \"kubernetes.io/projected/c69ee1a0-f4ea-4f73-a53a-e435c99524bb-kube-api-access-9fv75\") pod \"payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9\" (UID: \"c69ee1a0-f4ea-4f73-a53a-e435c99524bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9"
Sep 18 13:37:00.617313 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.617192    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69ee1a0-f4ea-4f73-a53a-e435c99524bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9\" (UID: \"c69ee1a0-f4ea-4f73-a53a-e435c99524bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9"
Sep 18 13:37:00.617860 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.617840    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69ee1a0-f4ea-4f73-a53a-e435c99524bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9\" (UID: \"c69ee1a0-f4ea-4f73-a53a-e435c99524bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9"
Sep 18 13:37:00.635462 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.635432    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fv75\" (UniqueName: \"kubernetes.io/projected/c69ee1a0-f4ea-4f73-a53a-e435c99524bb-kube-api-access-9fv75\") pod \"payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9\" (UID: \"c69ee1a0-f4ea-4f73-a53a-e435c99524bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9"
Sep 18 13:37:00.652941 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.652913    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp"
Sep 18 13:37:00.761621 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.761560    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9"
Sep 18 13:37:00.926549 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:00.926516    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp"]
Sep 18 13:37:00.928138 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:37:00.928112    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558b86b8_b911_4ce8_adaa_4b4af2f6833e.slice/crio-c25faacb5b4944d5cf395edf2c88002e4e697f3ba36a75170167f1b4d59b57cb WatchSource:0}: Error finding container c25faacb5b4944d5cf395edf2c88002e4e697f3ba36a75170167f1b4d59b57cb: Status 404 returned error can't find the container with id c25faacb5b4944d5cf395edf2c88002e4e697f3ba36a75170167f1b4d59b57cb
Sep 18 13:37:01.036937 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:01.036909    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9"]
Sep 18 13:37:01.038880 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:37:01.038848    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc69ee1a0_f4ea_4f73_a53a_e435c99524bb.slice/crio-f5ef78929b2d396f7734bbde973fcd5ff14347ad55b1a144c3b5d512f4689aa7 WatchSource:0}: Error finding container f5ef78929b2d396f7734bbde973fcd5ff14347ad55b1a144c3b5d512f4689aa7: Status 404 returned error can't find the container with id f5ef78929b2d396f7734bbde973fcd5ff14347ad55b1a144c3b5d512f4689aa7
Sep 18 13:37:01.210205 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:01.210099    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9" event={"ID":"c69ee1a0-f4ea-4f73-a53a-e435c99524bb","Type":"ContainerStarted","Data":"4dce847a6fb991d583b478316e46aa510405e6dc15f4322c859ce5433b0dd38e"}
Sep 18 13:37:01.210205 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:01.210168    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9"
Sep 18 13:37:01.210205 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:01.210196    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9" event={"ID":"c69ee1a0-f4ea-4f73-a53a-e435c99524bb","Type":"ContainerStarted","Data":"f5ef78929b2d396f7734bbde973fcd5ff14347ad55b1a144c3b5d512f4689aa7"}
Sep 18 13:37:01.211753 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:01.211727    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp" event={"ID":"558b86b8-b911-4ce8-adaa-4b4af2f6833e","Type":"ContainerStarted","Data":"9a54553ef9cd1564ac6f148ded3e96f9a6d2b83cc56fbe9ee57d88b90d3ccde0"}
Sep 18 13:37:01.211852 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:01.211773    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp"
Sep 18 13:37:01.211852 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:01.211790    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp" event={"ID":"558b86b8-b911-4ce8-adaa-4b4af2f6833e","Type":"ContainerStarted","Data":"c25faacb5b4944d5cf395edf2c88002e4e697f3ba36a75170167f1b4d59b57cb"}
Sep 18 13:37:01.239199 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:01.239143    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9" podStartSLOduration=1.239125338 podStartE2EDuration="1.239125338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:37:01.237989469 +0000 UTC m=+1329.672865875" watchObservedRunningTime="2026-09-18 13:37:01.239125338 +0000 UTC m=+1329.674001741"
Sep 18 13:37:01.280782 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:01.280724    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp" podStartSLOduration=1.280702903 podStartE2EDuration="1.280702903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:37:01.278366129 +0000 UTC m=+1329.713242533" watchObservedRunningTime="2026-09-18 13:37:01.280702903 +0000 UTC m=+1329.715579309"
Sep 18 13:37:12.220234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.220198    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp"
Sep 18 13:37:12.220624 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.220358    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9"
Sep 18 13:37:12.586655 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.586570    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"]
Sep 18 13:37:12.590257 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.590241    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:37:12.593182 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.593160    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-951469\"/\"distinct-2-w0-951469-kserve-self-signed-certs\""
Sep 18 13:37:12.608167 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.608141    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"]
Sep 18 13:37:12.737248 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.737203    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a8735a2-d52e-4781-bff2-6b516ecf5cec-home\") pod \"distinct-2-w0-951469-kserve-77d78557b6-pjmbq\" (UID: \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\") " pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:37:12.737452 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.737391    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a8735a2-d52e-4781-bff2-6b516ecf5cec-tmp-dir\") pod \"distinct-2-w0-951469-kserve-77d78557b6-pjmbq\" (UID: \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\") " pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:37:12.737551 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.737456    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5jsq\" (UniqueName: \"kubernetes.io/projected/9a8735a2-d52e-4781-bff2-6b516ecf5cec-kube-api-access-f5jsq\") pod \"distinct-2-w0-951469-kserve-77d78557b6-pjmbq\" (UID: \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\") " pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:37:12.737551 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.737490    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a8735a2-d52e-4781-bff2-6b516ecf5cec-model-cache\") pod \"distinct-2-w0-951469-kserve-77d78557b6-pjmbq\" (UID: \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\") " pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:37:12.737551 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.737527    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a8735a2-d52e-4781-bff2-6b516ecf5cec-tls-certs\") pod \"distinct-2-w0-951469-kserve-77d78557b6-pjmbq\" (UID: \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\") " pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:37:12.737744 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.737590    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a8735a2-d52e-4781-bff2-6b516ecf5cec-dshm\") pod \"distinct-2-w0-951469-kserve-77d78557b6-pjmbq\" (UID: \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\") " pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:37:12.839002 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.838906    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a8735a2-d52e-4781-bff2-6b516ecf5cec-tmp-dir\") pod \"distinct-2-w0-951469-kserve-77d78557b6-pjmbq\" (UID: \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\") " pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:37:12.839002 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.838975    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5jsq\" (UniqueName: \"kubernetes.io/projected/9a8735a2-d52e-4781-bff2-6b516ecf5cec-kube-api-access-f5jsq\") pod \"distinct-2-w0-951469-kserve-77d78557b6-pjmbq\" (UID: \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\") " pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:37:12.839194 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.839010    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a8735a2-d52e-4781-bff2-6b516ecf5cec-model-cache\") pod \"distinct-2-w0-951469-kserve-77d78557b6-pjmbq\" (UID: \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\") " pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:37:12.839194 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.839050    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a8735a2-d52e-4781-bff2-6b516ecf5cec-tls-certs\") pod \"distinct-2-w0-951469-kserve-77d78557b6-pjmbq\" (UID: \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\") " pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:37:12.839194 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.839115    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a8735a2-d52e-4781-bff2-6b516ecf5cec-dshm\") pod \"distinct-2-w0-951469-kserve-77d78557b6-pjmbq\" (UID: \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\") " pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:37:12.839194 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.839173    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a8735a2-d52e-4781-bff2-6b516ecf5cec-home\") pod \"distinct-2-w0-951469-kserve-77d78557b6-pjmbq\" (UID: \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\") " pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:37:12.839420 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.839394    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a8735a2-d52e-4781-bff2-6b516ecf5cec-model-cache\") pod \"distinct-2-w0-951469-kserve-77d78557b6-pjmbq\" (UID: \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\") " pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:37:12.839527 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.839504    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a8735a2-d52e-4781-bff2-6b516ecf5cec-tmp-dir\") pod \"distinct-2-w0-951469-kserve-77d78557b6-pjmbq\" (UID: \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\") " pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:37:12.839653 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.839565    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a8735a2-d52e-4781-bff2-6b516ecf5cec-home\") pod \"distinct-2-w0-951469-kserve-77d78557b6-pjmbq\" (UID: \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\") " pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:37:12.841460 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.841439    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a8735a2-d52e-4781-bff2-6b516ecf5cec-dshm\") pod \"distinct-2-w0-951469-kserve-77d78557b6-pjmbq\" (UID: \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\") " pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:37:12.841690 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.841674    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a8735a2-d52e-4781-bff2-6b516ecf5cec-tls-certs\") pod \"distinct-2-w0-951469-kserve-77d78557b6-pjmbq\" (UID: \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\") " pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:37:12.849656 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.849632    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5jsq\" (UniqueName: \"kubernetes.io/projected/9a8735a2-d52e-4781-bff2-6b516ecf5cec-kube-api-access-f5jsq\") pod \"distinct-2-w0-951469-kserve-77d78557b6-pjmbq\" (UID: \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\") " pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:37:12.900423 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:12.900396    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:37:13.177555 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:13.177526    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"]
Sep 18 13:37:13.179473 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:37:13.179447    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a8735a2_d52e_4781_bff2_6b516ecf5cec.slice/crio-8f2aa240584b16c8d4d6d79a2b3a7422a9d199b4a16a002b5b7756f1f4927139 WatchSource:0}: Error finding container 8f2aa240584b16c8d4d6d79a2b3a7422a9d199b4a16a002b5b7756f1f4927139: Status 404 returned error can't find the container with id 8f2aa240584b16c8d4d6d79a2b3a7422a9d199b4a16a002b5b7756f1f4927139
Sep 18 13:37:13.284664 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:13.284629    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq" event={"ID":"9a8735a2-d52e-4781-bff2-6b516ecf5cec","Type":"ContainerStarted","Data":"00f8e80fcccc457c2cf7f3784e5741e94157cbbd74cbc4e6b924882fa498159f"}
Sep 18 13:37:13.285076 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:13.284676    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq" event={"ID":"9a8735a2-d52e-4781-bff2-6b516ecf5cec","Type":"ContainerStarted","Data":"8f2aa240584b16c8d4d6d79a2b3a7422a9d199b4a16a002b5b7756f1f4927139"}
Sep 18 13:37:13.311068 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:13.310999    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq" podStartSLOduration=1.310982491 podStartE2EDuration="1.310982491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:37:13.308161571 +0000 UTC m=+1341.743037974" watchObservedRunningTime="2026-09-18 13:37:13.310982491 +0000 UTC m=+1341.745858895"
Sep 18 13:37:22.901433 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:22.901391    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:37:22.901954 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:22.901545    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:37:22.914118 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:22.914094    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:37:23.358713 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:23.358682    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:37:31.402795 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:31.402757    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb"]
Sep 18 13:37:31.409703 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:31.409675    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb"
Sep 18 13:37:31.428096 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:31.428062    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb\" (UID: \"d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb"
Sep 18 13:37:31.428096 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:31.428095    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5kpj\" (UniqueName: \"kubernetes.io/projected/d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b-kube-api-access-g5kpj\") pod \"payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb\" (UID: \"d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb"
Sep 18 13:37:31.433395 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:31.433367    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-c50e10\""
Sep 18 13:37:31.433395 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:31.433370    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-c50e10-dockercfg-rjv99\""
Sep 18 13:37:31.446236 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:37:31.446200    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb\" is forbidden: User \"system:node:ip-10-0-129-234.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-129-234.ec2.internal' and this object" podUID="d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb"
Sep 18 13:37:31.458874 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:31.458837    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb"]
Sep 18 13:37:31.528670 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:31.528636    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb\" (UID: \"d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb"
Sep 18 13:37:31.528670 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:31.528673    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5kpj\" (UniqueName: \"kubernetes.io/projected/d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b-kube-api-access-g5kpj\") pod \"payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb\" (UID: \"d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb"
Sep 18 13:37:31.529344 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:31.529324    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb\" (UID: \"d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb"
Sep 18 13:37:31.553156 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:31.553127    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5kpj\" (UniqueName: \"kubernetes.io/projected/d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b-kube-api-access-g5kpj\") pod \"payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb\" (UID: \"d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb"
Sep 18 13:37:31.729188 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:31.729099    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb"
Sep 18 13:37:32.015170 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:32.015131    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb"]
Sep 18 13:37:32.017364 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:37:32.017330    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f7ad61_fa71_4bbe_93c0_f10f7d5a871b.slice/crio-bdba9f2fb8a93afb8bd9d1fbf7a5f42ed307a3000cbdb87544138018d6fbb3f8 WatchSource:0}: Error finding container bdba9f2fb8a93afb8bd9d1fbf7a5f42ed307a3000cbdb87544138018d6fbb3f8: Status 404 returned error can't find the container with id bdba9f2fb8a93afb8bd9d1fbf7a5f42ed307a3000cbdb87544138018d6fbb3f8
Sep 18 13:37:32.402334 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:32.402297    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb" event={"ID":"d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b","Type":"ContainerStarted","Data":"aa778e157082ef1a68e4294f41ba78865eadeb40440794217e276fec59af76da"}
Sep 18 13:37:32.402543 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:32.402359    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb" event={"ID":"d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b","Type":"ContainerStarted","Data":"bdba9f2fb8a93afb8bd9d1fbf7a5f42ed307a3000cbdb87544138018d6fbb3f8"}
Sep 18 13:37:32.402543 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:32.402434    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb"
Sep 18 13:37:32.430032 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:32.429981    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb" podStartSLOduration=1.429965273 podStartE2EDuration="1.429965273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:37:32.427775092 +0000 UTC m=+1360.862651496" watchObservedRunningTime="2026-09-18 13:37:32.429965273 +0000 UTC m=+1360.864841677"
Sep 18 13:37:43.410809 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:43.410773    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb"
Sep 18 13:37:52.232457 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:52.232426    2566 scope.go:117] "RemoveContainer" containerID="e2756b62618944cdb428c37b2a9a2fb04252fb80059cdf2dff6cd528a95ff37c"
Sep 18 13:37:52.241590 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:52.241567    2566 scope.go:117] "RemoveContainer" containerID="2112ebc900f496725ba3c30b39c0161ea895792d228f37471bac3fbc4373df04"
Sep 18 13:37:52.249777 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:37:52.249758    2566 scope.go:117] "RemoveContainer" containerID="0a0a4e429152d9a508a205da253d92368c54e2814827a7f41b9b1264ac6e1d08"
Sep 18 13:38:50.927621 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:38:50.927514    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"]
Sep 18 13:39:04.742224 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:04.742179    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"]
Sep 18 13:39:05.957159 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:05.957088    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf" podUID="296a73a3-b94f-44d7-949c-a38b18c8f66d" containerName="main" containerID="cri-o://7af492cd718602ae6555dc65640a463db2582622d28f42a4d555bcbfe40cfa14" gracePeriod=45
Sep 18 13:39:06.212144 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.212079    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:39:06.306783 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.306738    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/296a73a3-b94f-44d7-949c-a38b18c8f66d-tls-certs\") pod \"296a73a3-b94f-44d7-949c-a38b18c8f66d\" (UID: \"296a73a3-b94f-44d7-949c-a38b18c8f66d\") "
Sep 18 13:39:06.306996 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.306807    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/296a73a3-b94f-44d7-949c-a38b18c8f66d-model-cache\") pod \"296a73a3-b94f-44d7-949c-a38b18c8f66d\" (UID: \"296a73a3-b94f-44d7-949c-a38b18c8f66d\") "
Sep 18 13:39:06.306996 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.306847    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/296a73a3-b94f-44d7-949c-a38b18c8f66d-tmp-dir\") pod \"296a73a3-b94f-44d7-949c-a38b18c8f66d\" (UID: \"296a73a3-b94f-44d7-949c-a38b18c8f66d\") "
Sep 18 13:39:06.306996 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.306874    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbpcp\" (UniqueName: \"kubernetes.io/projected/296a73a3-b94f-44d7-949c-a38b18c8f66d-kube-api-access-jbpcp\") pod \"296a73a3-b94f-44d7-949c-a38b18c8f66d\" (UID: \"296a73a3-b94f-44d7-949c-a38b18c8f66d\") "
Sep 18 13:39:06.306996 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.306915    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/296a73a3-b94f-44d7-949c-a38b18c8f66d-dshm\") pod \"296a73a3-b94f-44d7-949c-a38b18c8f66d\" (UID: \"296a73a3-b94f-44d7-949c-a38b18c8f66d\") "
Sep 18 13:39:06.306996 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.306967    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/296a73a3-b94f-44d7-949c-a38b18c8f66d-home\") pod \"296a73a3-b94f-44d7-949c-a38b18c8f66d\" (UID: \"296a73a3-b94f-44d7-949c-a38b18c8f66d\") "
Sep 18 13:39:06.307244 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.307096    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/296a73a3-b94f-44d7-949c-a38b18c8f66d-model-cache" (OuterVolumeSpecName: "model-cache") pod "296a73a3-b94f-44d7-949c-a38b18c8f66d" (UID: "296a73a3-b94f-44d7-949c-a38b18c8f66d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:39:06.307244 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.307109    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/296a73a3-b94f-44d7-949c-a38b18c8f66d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "296a73a3-b94f-44d7-949c-a38b18c8f66d" (UID: "296a73a3-b94f-44d7-949c-a38b18c8f66d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:39:06.307330 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.307256    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/296a73a3-b94f-44d7-949c-a38b18c8f66d-home" (OuterVolumeSpecName: "home") pod "296a73a3-b94f-44d7-949c-a38b18c8f66d" (UID: "296a73a3-b94f-44d7-949c-a38b18c8f66d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:39:06.307397 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.307379    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/296a73a3-b94f-44d7-949c-a38b18c8f66d-model-cache\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:39:06.307435 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.307404    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/296a73a3-b94f-44d7-949c-a38b18c8f66d-tmp-dir\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:39:06.307435 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.307416    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/296a73a3-b94f-44d7-949c-a38b18c8f66d-home\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:39:06.309186 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.309156    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/296a73a3-b94f-44d7-949c-a38b18c8f66d-dshm" (OuterVolumeSpecName: "dshm") pod "296a73a3-b94f-44d7-949c-a38b18c8f66d" (UID: "296a73a3-b94f-44d7-949c-a38b18c8f66d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:39:06.309310 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.309199    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/296a73a3-b94f-44d7-949c-a38b18c8f66d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "296a73a3-b94f-44d7-949c-a38b18c8f66d" (UID: "296a73a3-b94f-44d7-949c-a38b18c8f66d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:39:06.309310 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.309212    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/296a73a3-b94f-44d7-949c-a38b18c8f66d-kube-api-access-jbpcp" (OuterVolumeSpecName: "kube-api-access-jbpcp") pod "296a73a3-b94f-44d7-949c-a38b18c8f66d" (UID: "296a73a3-b94f-44d7-949c-a38b18c8f66d"). InnerVolumeSpecName "kube-api-access-jbpcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:39:06.408041 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.408000    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbpcp\" (UniqueName: \"kubernetes.io/projected/296a73a3-b94f-44d7-949c-a38b18c8f66d-kube-api-access-jbpcp\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:39:06.408041 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.408034    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/296a73a3-b94f-44d7-949c-a38b18c8f66d-dshm\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:39:06.408041 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.408043    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/296a73a3-b94f-44d7-949c-a38b18c8f66d-tls-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:39:06.942308 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.942269    2566 generic.go:356] "Generic (PLEG): container finished" podID="296a73a3-b94f-44d7-949c-a38b18c8f66d" containerID="7af492cd718602ae6555dc65640a463db2582622d28f42a4d555bcbfe40cfa14" exitCode=0
Sep 18 13:39:06.942480 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.942318    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"
Sep 18 13:39:06.942480 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.942354    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf" event={"ID":"296a73a3-b94f-44d7-949c-a38b18c8f66d","Type":"ContainerDied","Data":"7af492cd718602ae6555dc65640a463db2582622d28f42a4d555bcbfe40cfa14"}
Sep 18 13:39:06.942480 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.942402    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf" event={"ID":"296a73a3-b94f-44d7-949c-a38b18c8f66d","Type":"ContainerDied","Data":"6f6e9bae21743cf497e8415756732bd2f9902a2f9d5699dbd0823e55e12419df"}
Sep 18 13:39:06.942480 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.942420    2566 scope.go:117] "RemoveContainer" containerID="7af492cd718602ae6555dc65640a463db2582622d28f42a4d555bcbfe40cfa14"
Sep 18 13:39:06.951826 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.951809    2566 scope.go:117] "RemoveContainer" containerID="7af492cd718602ae6555dc65640a463db2582622d28f42a4d555bcbfe40cfa14"
Sep 18 13:39:06.952115 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:39:06.952097    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af492cd718602ae6555dc65640a463db2582622d28f42a4d555bcbfe40cfa14\": container with ID starting with 7af492cd718602ae6555dc65640a463db2582622d28f42a4d555bcbfe40cfa14 not found: ID does not exist" containerID="7af492cd718602ae6555dc65640a463db2582622d28f42a4d555bcbfe40cfa14"
Sep 18 13:39:06.952159 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.952125    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af492cd718602ae6555dc65640a463db2582622d28f42a4d555bcbfe40cfa14"} err="failed to get container status \"7af492cd718602ae6555dc65640a463db2582622d28f42a4d555bcbfe40cfa14\": rpc error: code = NotFound desc = could not find container \"7af492cd718602ae6555dc65640a463db2582622d28f42a4d555bcbfe40cfa14\": container with ID starting with 7af492cd718602ae6555dc65640a463db2582622d28f42a4d555bcbfe40cfa14 not found: ID does not exist"
Sep 18 13:39:06.966706 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.966671    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"]
Sep 18 13:39:06.973895 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:06.973868    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-b-158c7039-kserve-5849dc7899-mr5jf"]
Sep 18 13:39:08.085150 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:08.085111    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="296a73a3-b94f-44d7-949c-a38b18c8f66d" path="/var/lib/kubelet/pods/296a73a3-b94f-44d7-949c-a38b18c8f66d/volumes"
Sep 18 13:39:18.077284 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:18.077251    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa9905fd-zkctw"]
Sep 18 13:39:18.077951 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:18.077679    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="296a73a3-b94f-44d7-949c-a38b18c8f66d" containerName="main"
Sep 18 13:39:18.077951 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:18.077692    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="296a73a3-b94f-44d7-949c-a38b18c8f66d" containerName="main"
Sep 18 13:39:18.077951 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:18.077779    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="296a73a3-b94f-44d7-949c-a38b18c8f66d" containerName="main"
Sep 18 13:39:18.081045 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:18.081026    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa9905fd-zkctw"
Sep 18 13:39:18.083685 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:18.083653    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nh2cd\""
Sep 18 13:39:18.084716 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:18.084694    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:39:18.087362 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:18.087338    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa9905fd-zkctw"]
Sep 18 13:39:18.127217 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:18.127176    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a3930f-4286-455a-9518-8071ccad4d7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fa9905fd-zkctw\" (UID: \"93a3930f-4286-455a-9518-8071ccad4d7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa9905fd-zkctw"
Sep 18 13:39:18.228395 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:18.228363    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a3930f-4286-455a-9518-8071ccad4d7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fa9905fd-zkctw\" (UID: \"93a3930f-4286-455a-9518-8071ccad4d7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa9905fd-zkctw"
Sep 18 13:39:18.229045 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:18.229025    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a3930f-4286-455a-9518-8071ccad4d7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fa9905fd-zkctw\" (UID: \"93a3930f-4286-455a-9518-8071ccad4d7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa9905fd-zkctw"
Sep 18 13:39:18.393545 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:18.393509    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa9905fd-zkctw"
Sep 18 13:39:18.673713 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:18.673680    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa9905fd-zkctw"]
Sep 18 13:39:18.675722 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:39:18.675694    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a3930f_4286_455a_9518_8071ccad4d7c.slice/crio-4d451a92b815e0f51a1d320860942be59c9211adaf0eae468d7d1542f1de0017 WatchSource:0}: Error finding container 4d451a92b815e0f51a1d320860942be59c9211adaf0eae468d7d1542f1de0017: Status 404 returned error can't find the container with id 4d451a92b815e0f51a1d320860942be59c9211adaf0eae468d7d1542f1de0017
Sep 18 13:39:19.013442 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:19.013409    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa9905fd-zkctw" event={"ID":"93a3930f-4286-455a-9518-8071ccad4d7c","Type":"ContainerStarted","Data":"5e032634f5925d03b4ac978af1cc5b9a4debac1bbdc4c0d8a33787ba88873426"}
Sep 18 13:39:19.013650 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:19.013467    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa9905fd-zkctw" event={"ID":"93a3930f-4286-455a-9518-8071ccad4d7c","Type":"ContainerStarted","Data":"4d451a92b815e0f51a1d320860942be59c9211adaf0eae468d7d1542f1de0017"}
Sep 18 13:39:19.038709 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:19.038648    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa9905fd-zkctw" podStartSLOduration=1.038629085 podStartE2EDuration="1.038629085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:39:19.038071532 +0000 UTC m=+1467.472947937" watchObservedRunningTime="2026-09-18 13:39:19.038629085 +0000 UTC m=+1467.473505488"
Sep 18 13:39:19.761762 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:19.761677    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5" podUID="69b4b46b-eee4-49fc-8165-b9b6d664535f" containerName="main" containerID="cri-o://bcef7297356f46ba3106bd00d6a5e9cacb0f19593d21a938b18437ffdf5ee2a7" gracePeriod=45
Sep 18 13:39:20.021234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.021161    2566 generic.go:356] "Generic (PLEG): container finished" podID="69b4b46b-eee4-49fc-8165-b9b6d664535f" containerID="bcef7297356f46ba3106bd00d6a5e9cacb0f19593d21a938b18437ffdf5ee2a7" exitCode=0
Sep 18 13:39:20.021371 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.021230    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5" event={"ID":"69b4b46b-eee4-49fc-8165-b9b6d664535f","Type":"ContainerDied","Data":"bcef7297356f46ba3106bd00d6a5e9cacb0f19593d21a938b18437ffdf5ee2a7"}
Sep 18 13:39:20.021371 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.021278    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5" event={"ID":"69b4b46b-eee4-49fc-8165-b9b6d664535f","Type":"ContainerDied","Data":"b72acb8cbaafe2f9cb218b01db11cf6e3360ed0827e7b99bcd2886f7b987aca7"}
Sep 18 13:39:20.021371 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.021295    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b72acb8cbaafe2f9cb218b01db11cf6e3360ed0827e7b99bcd2886f7b987aca7"
Sep 18 13:39:20.022864 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.022841    2566 generic.go:356] "Generic (PLEG): container finished" podID="93a3930f-4286-455a-9518-8071ccad4d7c" containerID="5e032634f5925d03b4ac978af1cc5b9a4debac1bbdc4c0d8a33787ba88873426" exitCode=0
Sep 18 13:39:20.022984 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.022932    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa9905fd-zkctw" event={"ID":"93a3930f-4286-455a-9518-8071ccad4d7c","Type":"ContainerDied","Data":"5e032634f5925d03b4ac978af1cc5b9a4debac1bbdc4c0d8a33787ba88873426"}
Sep 18 13:39:20.023430 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.023415    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:39:20.147437 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.147401    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69b4b46b-eee4-49fc-8165-b9b6d664535f-home\") pod \"69b4b46b-eee4-49fc-8165-b9b6d664535f\" (UID: \"69b4b46b-eee4-49fc-8165-b9b6d664535f\") "
Sep 18 13:39:20.147642 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.147456    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69b4b46b-eee4-49fc-8165-b9b6d664535f-tls-certs\") pod \"69b4b46b-eee4-49fc-8165-b9b6d664535f\" (UID: \"69b4b46b-eee4-49fc-8165-b9b6d664535f\") "
Sep 18 13:39:20.147642 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.147521    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69b4b46b-eee4-49fc-8165-b9b6d664535f-tmp-dir\") pod \"69b4b46b-eee4-49fc-8165-b9b6d664535f\" (UID: \"69b4b46b-eee4-49fc-8165-b9b6d664535f\") "
Sep 18 13:39:20.147642 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.147538    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69b4b46b-eee4-49fc-8165-b9b6d664535f-dshm\") pod \"69b4b46b-eee4-49fc-8165-b9b6d664535f\" (UID: \"69b4b46b-eee4-49fc-8165-b9b6d664535f\") "
Sep 18 13:39:20.147642 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.147564    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69b4b46b-eee4-49fc-8165-b9b6d664535f-model-cache\") pod \"69b4b46b-eee4-49fc-8165-b9b6d664535f\" (UID: \"69b4b46b-eee4-49fc-8165-b9b6d664535f\") "
Sep 18 13:39:20.147642 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.147621    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxxsn\" (UniqueName: \"kubernetes.io/projected/69b4b46b-eee4-49fc-8165-b9b6d664535f-kube-api-access-rxxsn\") pod \"69b4b46b-eee4-49fc-8165-b9b6d664535f\" (UID: \"69b4b46b-eee4-49fc-8165-b9b6d664535f\") "
Sep 18 13:39:20.147931 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.147781    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b4b46b-eee4-49fc-8165-b9b6d664535f-home" (OuterVolumeSpecName: "home") pod "69b4b46b-eee4-49fc-8165-b9b6d664535f" (UID: "69b4b46b-eee4-49fc-8165-b9b6d664535f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:39:20.147931 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.147832    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b4b46b-eee4-49fc-8165-b9b6d664535f-model-cache" (OuterVolumeSpecName: "model-cache") pod "69b4b46b-eee4-49fc-8165-b9b6d664535f" (UID: "69b4b46b-eee4-49fc-8165-b9b6d664535f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:39:20.147931 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.147865    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b4b46b-eee4-49fc-8165-b9b6d664535f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "69b4b46b-eee4-49fc-8165-b9b6d664535f" (UID: "69b4b46b-eee4-49fc-8165-b9b6d664535f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:39:20.148148 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.148016    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69b4b46b-eee4-49fc-8165-b9b6d664535f-tmp-dir\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:39:20.148148 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.148033    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69b4b46b-eee4-49fc-8165-b9b6d664535f-model-cache\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:39:20.148148 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.148041    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69b4b46b-eee4-49fc-8165-b9b6d664535f-home\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:39:20.149790 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.149761    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b4b46b-eee4-49fc-8165-b9b6d664535f-dshm" (OuterVolumeSpecName: "dshm") pod "69b4b46b-eee4-49fc-8165-b9b6d664535f" (UID: "69b4b46b-eee4-49fc-8165-b9b6d664535f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:39:20.149914 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.149854    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b4b46b-eee4-49fc-8165-b9b6d664535f-kube-api-access-rxxsn" (OuterVolumeSpecName: "kube-api-access-rxxsn") pod "69b4b46b-eee4-49fc-8165-b9b6d664535f" (UID: "69b4b46b-eee4-49fc-8165-b9b6d664535f"). InnerVolumeSpecName "kube-api-access-rxxsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:39:20.149914 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.149867    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69b4b46b-eee4-49fc-8165-b9b6d664535f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69b4b46b-eee4-49fc-8165-b9b6d664535f" (UID: "69b4b46b-eee4-49fc-8165-b9b6d664535f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:39:20.248896 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.248859    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69b4b46b-eee4-49fc-8165-b9b6d664535f-tls-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:39:20.248896 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.248886    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69b4b46b-eee4-49fc-8165-b9b6d664535f-dshm\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:39:20.248896 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:20.248896    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxxsn\" (UniqueName: \"kubernetes.io/projected/69b4b46b-eee4-49fc-8165-b9b6d664535f-kube-api-access-rxxsn\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:39:21.034744 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.034709    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"
Sep 18 13:39:21.066071 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.066029    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"]
Sep 18 13:39:21.068298 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.068267    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/e2e-conflict-a-158c7039-kserve-6ccb589b4f-rcgt5"]
Sep 18 13:39:21.180708 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.180685    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa9905fd-zkctw"
Sep 18 13:39:21.258378 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.258347    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a3930f-4286-455a-9518-8071ccad4d7c-maas-api-service-ca\") pod \"93a3930f-4286-455a-9518-8071ccad4d7c\" (UID: \"93a3930f-4286-455a-9518-8071ccad4d7c\") "
Sep 18 13:39:21.258771 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.258750    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a3930f-4286-455a-9518-8071ccad4d7c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "93a3930f-4286-455a-9518-8071ccad4d7c" (UID: "93a3930f-4286-455a-9518-8071ccad4d7c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:39:21.359848 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.359812    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a3930f-4286-455a-9518-8071ccad4d7c-maas-api-service-ca\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:39:21.787078 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.786993    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"]
Sep 18 13:39:21.787434 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.787421    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93a3930f-4286-455a-9518-8071ccad4d7c" containerName="revoke-keys"
Sep 18 13:39:21.787478 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.787435    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="93a3930f-4286-455a-9518-8071ccad4d7c" containerName="revoke-keys"
Sep 18 13:39:21.787478 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.787460    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69b4b46b-eee4-49fc-8165-b9b6d664535f" containerName="main"
Sep 18 13:39:21.787478 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.787466    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="69b4b46b-eee4-49fc-8165-b9b6d664535f" containerName="main"
Sep 18 13:39:21.787565 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.787525    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="69b4b46b-eee4-49fc-8165-b9b6d664535f" containerName="main"
Sep 18 13:39:21.787565 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.787534    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="93a3930f-4286-455a-9518-8071ccad4d7c" containerName="revoke-keys"
Sep 18 13:39:21.790511 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.790492    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:39:21.796697 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.796630    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5be8a3\"/\"distinct-w5-5be8a3-kserve-self-signed-certs\""
Sep 18 13:39:21.796697 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.796669    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-5be8a3\"/\"kube-root-ca.crt\""
Sep 18 13:39:21.796905 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.796763    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-5be8a3\"/\"openshift-service-ca.crt\""
Sep 18 13:39:21.796975 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.796949    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5be8a3\"/\"default-dockercfg-b7jf8\""
Sep 18 13:39:21.804999 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.804976    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"]
Sep 18 13:39:21.864376 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.864338    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16906812-0e34-470c-b4bd-30f5e61d3f9b-tmp-dir\") pod \"distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w\" (UID: \"16906812-0e34-470c-b4bd-30f5e61d3f9b\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:39:21.864554 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.864432    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16906812-0e34-470c-b4bd-30f5e61d3f9b-model-cache\") pod \"distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w\" (UID: \"16906812-0e34-470c-b4bd-30f5e61d3f9b\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:39:21.864554 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.864463    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16906812-0e34-470c-b4bd-30f5e61d3f9b-dshm\") pod \"distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w\" (UID: \"16906812-0e34-470c-b4bd-30f5e61d3f9b\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:39:21.864554 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.864483    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16906812-0e34-470c-b4bd-30f5e61d3f9b-tls-certs\") pod \"distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w\" (UID: \"16906812-0e34-470c-b4bd-30f5e61d3f9b\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:39:21.864733 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.864581    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16906812-0e34-470c-b4bd-30f5e61d3f9b-home\") pod \"distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w\" (UID: \"16906812-0e34-470c-b4bd-30f5e61d3f9b\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:39:21.864733 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.864621    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh9zj\" (UniqueName: \"kubernetes.io/projected/16906812-0e34-470c-b4bd-30f5e61d3f9b-kube-api-access-zh9zj\") pod \"distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w\" (UID: \"16906812-0e34-470c-b4bd-30f5e61d3f9b\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:39:21.965539 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.965504    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16906812-0e34-470c-b4bd-30f5e61d3f9b-home\") pod \"distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w\" (UID: \"16906812-0e34-470c-b4bd-30f5e61d3f9b\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:39:21.965773 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.965550    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh9zj\" (UniqueName: \"kubernetes.io/projected/16906812-0e34-470c-b4bd-30f5e61d3f9b-kube-api-access-zh9zj\") pod \"distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w\" (UID: \"16906812-0e34-470c-b4bd-30f5e61d3f9b\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:39:21.965773 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.965577    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16906812-0e34-470c-b4bd-30f5e61d3f9b-tmp-dir\") pod \"distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w\" (UID: \"16906812-0e34-470c-b4bd-30f5e61d3f9b\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:39:21.965773 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.965654    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16906812-0e34-470c-b4bd-30f5e61d3f9b-model-cache\") pod \"distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w\" (UID: \"16906812-0e34-470c-b4bd-30f5e61d3f9b\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:39:21.965773 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.965673    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16906812-0e34-470c-b4bd-30f5e61d3f9b-dshm\") pod \"distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w\" (UID: \"16906812-0e34-470c-b4bd-30f5e61d3f9b\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:39:21.966007 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.965805    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16906812-0e34-470c-b4bd-30f5e61d3f9b-tls-certs\") pod \"distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w\" (UID: \"16906812-0e34-470c-b4bd-30f5e61d3f9b\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:39:21.966048 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.966003    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16906812-0e34-470c-b4bd-30f5e61d3f9b-home\") pod \"distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w\" (UID: \"16906812-0e34-470c-b4bd-30f5e61d3f9b\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:39:21.966086 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.966050    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16906812-0e34-470c-b4bd-30f5e61d3f9b-model-cache\") pod \"distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w\" (UID: \"16906812-0e34-470c-b4bd-30f5e61d3f9b\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:39:21.966122 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.966105    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16906812-0e34-470c-b4bd-30f5e61d3f9b-tmp-dir\") pod \"distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w\" (UID: \"16906812-0e34-470c-b4bd-30f5e61d3f9b\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:39:21.967906 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.967884    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16906812-0e34-470c-b4bd-30f5e61d3f9b-dshm\") pod \"distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w\" (UID: \"16906812-0e34-470c-b4bd-30f5e61d3f9b\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:39:21.968300 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.968275    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16906812-0e34-470c-b4bd-30f5e61d3f9b-tls-certs\") pod \"distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w\" (UID: \"16906812-0e34-470c-b4bd-30f5e61d3f9b\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:39:21.980464 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:21.980443    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh9zj\" (UniqueName: \"kubernetes.io/projected/16906812-0e34-470c-b4bd-30f5e61d3f9b-kube-api-access-zh9zj\") pod \"distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w\" (UID: \"16906812-0e34-470c-b4bd-30f5e61d3f9b\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:39:22.042056 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:22.041969    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa9905fd-zkctw"
Sep 18 13:39:22.042056 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:22.042019    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa9905fd-zkctw" event={"ID":"93a3930f-4286-455a-9518-8071ccad4d7c","Type":"ContainerDied","Data":"4d451a92b815e0f51a1d320860942be59c9211adaf0eae468d7d1542f1de0017"}
Sep 18 13:39:22.042470 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:22.042071    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d451a92b815e0f51a1d320860942be59c9211adaf0eae468d7d1542f1de0017"
Sep 18 13:39:22.078039 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:22.078007    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69b4b46b-eee4-49fc-8165-b9b6d664535f" path="/var/lib/kubelet/pods/69b4b46b-eee4-49fc-8165-b9b6d664535f/volumes"
Sep 18 13:39:22.100655 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:22.100623    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:39:22.384881 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:22.384850    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"]
Sep 18 13:39:22.386331 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:39:22.386305    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16906812_0e34_470c_b4bd_30f5e61d3f9b.slice/crio-dd44a683aa5128f9cdc265607ccf8bf33f94f50ace02b6b2606b2641b7085405 WatchSource:0}: Error finding container dd44a683aa5128f9cdc265607ccf8bf33f94f50ace02b6b2606b2641b7085405: Status 404 returned error can't find the container with id dd44a683aa5128f9cdc265607ccf8bf33f94f50ace02b6b2606b2641b7085405
Sep 18 13:39:23.049469 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:23.049430    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w" event={"ID":"16906812-0e34-470c-b4bd-30f5e61d3f9b","Type":"ContainerStarted","Data":"1b3872acaabe5f18b9dc8f4b9aed5fc68321b5fe3a616fe7136a9a4cb237a007"}
Sep 18 13:39:23.049906 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:23.049486    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w" event={"ID":"16906812-0e34-470c-b4bd-30f5e61d3f9b","Type":"ContainerStarted","Data":"dd44a683aa5128f9cdc265607ccf8bf33f94f50ace02b6b2606b2641b7085405"}
Sep 18 13:39:23.075351 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:23.075296    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w" podStartSLOduration=2.075279683 podStartE2EDuration="2.075279683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:39:23.074248713 +0000 UTC m=+1471.509125118" watchObservedRunningTime="2026-09-18 13:39:23.075279683 +0000 UTC m=+1471.510156131"
Sep 18 13:39:32.101008 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:32.100972    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:39:32.101564 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:32.101047    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:39:32.114019 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:32.113986    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:39:33.116952 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:33.116915    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:39:34.162697 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:34.162661    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc"]
Sep 18 13:39:34.163235 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:34.163178    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc" podUID="af436413-64b2-4248-b6ba-2ffd075df6c8" containerName="payload-pre-processing" containerID="cri-o://8690c8238c15599864b6764e0c333e48baae4ba4050a5c75f7cdf63f8ab6a779" gracePeriod=30
Sep 18 13:39:34.458294 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:34.458261    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc"
Sep 18 13:39:34.561968 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:34.561914    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa9905fd-zkctw"]
Sep 18 13:39:34.579278 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:34.579223    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fa9905fd-zkctw"]
Sep 18 13:39:34.591944 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:34.591914    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af436413-64b2-4248-b6ba-2ffd075df6c8-plugins-config-volume\") pod \"af436413-64b2-4248-b6ba-2ffd075df6c8\" (UID: \"af436413-64b2-4248-b6ba-2ffd075df6c8\") "
Sep 18 13:39:34.592051 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:34.591954    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ptd8\" (UniqueName: \"kubernetes.io/projected/af436413-64b2-4248-b6ba-2ffd075df6c8-kube-api-access-9ptd8\") pod \"af436413-64b2-4248-b6ba-2ffd075df6c8\" (UID: \"af436413-64b2-4248-b6ba-2ffd075df6c8\") "
Sep 18 13:39:34.592272 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:34.592248    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af436413-64b2-4248-b6ba-2ffd075df6c8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "af436413-64b2-4248-b6ba-2ffd075df6c8" (UID: "af436413-64b2-4248-b6ba-2ffd075df6c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:39:34.594119 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:34.594093    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af436413-64b2-4248-b6ba-2ffd075df6c8-kube-api-access-9ptd8" (OuterVolumeSpecName: "kube-api-access-9ptd8") pod "af436413-64b2-4248-b6ba-2ffd075df6c8" (UID: "af436413-64b2-4248-b6ba-2ffd075df6c8"). InnerVolumeSpecName "kube-api-access-9ptd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:39:34.693713 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:34.693623    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af436413-64b2-4248-b6ba-2ffd075df6c8-plugins-config-volume\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:39:34.693713 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:34.693655    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ptd8\" (UniqueName: \"kubernetes.io/projected/af436413-64b2-4248-b6ba-2ffd075df6c8-kube-api-access-9ptd8\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:39:35.118367 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:35.118329    2566 generic.go:356] "Generic (PLEG): container finished" podID="af436413-64b2-4248-b6ba-2ffd075df6c8" containerID="8690c8238c15599864b6764e0c333e48baae4ba4050a5c75f7cdf63f8ab6a779" exitCode=0
Sep 18 13:39:35.118570 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:35.118377    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc"
Sep 18 13:39:35.118570 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:35.118422    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc" event={"ID":"af436413-64b2-4248-b6ba-2ffd075df6c8","Type":"ContainerDied","Data":"8690c8238c15599864b6764e0c333e48baae4ba4050a5c75f7cdf63f8ab6a779"}
Sep 18 13:39:35.118570 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:35.118480    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc" event={"ID":"af436413-64b2-4248-b6ba-2ffd075df6c8","Type":"ContainerDied","Data":"c2d3611f60f18618278803edc0d9744651560d1b78189b9b345e024983844c3c"}
Sep 18 13:39:35.118570 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:35.118506    2566 scope.go:117] "RemoveContainer" containerID="8690c8238c15599864b6764e0c333e48baae4ba4050a5c75f7cdf63f8ab6a779"
Sep 18 13:39:35.128377 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:35.128357    2566 scope.go:117] "RemoveContainer" containerID="8690c8238c15599864b6764e0c333e48baae4ba4050a5c75f7cdf63f8ab6a779"
Sep 18 13:39:35.128674 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:39:35.128658    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8690c8238c15599864b6764e0c333e48baae4ba4050a5c75f7cdf63f8ab6a779\": container with ID starting with 8690c8238c15599864b6764e0c333e48baae4ba4050a5c75f7cdf63f8ab6a779 not found: ID does not exist" containerID="8690c8238c15599864b6764e0c333e48baae4ba4050a5c75f7cdf63f8ab6a779"
Sep 18 13:39:35.128728 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:35.128681    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8690c8238c15599864b6764e0c333e48baae4ba4050a5c75f7cdf63f8ab6a779"} err="failed to get container status \"8690c8238c15599864b6764e0c333e48baae4ba4050a5c75f7cdf63f8ab6a779\": rpc error: code = NotFound desc = could not find container \"8690c8238c15599864b6764e0c333e48baae4ba4050a5c75f7cdf63f8ab6a779\": container with ID starting with 8690c8238c15599864b6764e0c333e48baae4ba4050a5c75f7cdf63f8ab6a779 not found: ID does not exist"
Sep 18 13:39:35.183637 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:35.183604    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc"]
Sep 18 13:39:35.206353 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:35.206321    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fa9905fd-c74dcdc5d-r7pkc"]
Sep 18 13:39:36.078496 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:36.078459    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93a3930f-4286-455a-9518-8071ccad4d7c" path="/var/lib/kubelet/pods/93a3930f-4286-455a-9518-8071ccad4d7c/volumes"
Sep 18 13:39:36.078903 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:36.078887    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af436413-64b2-4248-b6ba-2ffd075df6c8" path="/var/lib/kubelet/pods/af436413-64b2-4248-b6ba-2ffd075df6c8/volumes"
Sep 18 13:39:46.078893 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.078840    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"]
Sep 18 13:39:46.079311 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.079297    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af436413-64b2-4248-b6ba-2ffd075df6c8" containerName="payload-pre-processing"
Sep 18 13:39:46.079361 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.079312    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="af436413-64b2-4248-b6ba-2ffd075df6c8" containerName="payload-pre-processing"
Sep 18 13:39:46.079411 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.079401    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="af436413-64b2-4248-b6ba-2ffd075df6c8" containerName="payload-pre-processing"
Sep 18 13:39:46.084183 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.084154    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:39:46.087887 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.087838    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c50e10\"/\"auth-test-model-c50e10-kserve-self-signed-certs\""
Sep 18 13:39:46.088055 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.087882    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c50e10\"/\"default-dockercfg-g82ls\""
Sep 18 13:39:46.088055 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.087914    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c50e10\"/\"openshift-service-ca.crt\""
Sep 18 13:39:46.088055 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.087872    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c50e10\"/\"kube-root-ca.crt\""
Sep 18 13:39:46.090716 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.090693    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"]
Sep 18 13:39:46.212490 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.212448    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e66f5234-dc9b-4fc6-9d16-c520f1a08142-home\") pod \"auth-test-model-c50e10-kserve-6df69ff995-f8gvd\" (UID: \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:39:46.212690 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.212514    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66f5234-dc9b-4fc6-9d16-c520f1a08142-model-cache\") pod \"auth-test-model-c50e10-kserve-6df69ff995-f8gvd\" (UID: \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:39:46.212690 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.212566    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e66f5234-dc9b-4fc6-9d16-c520f1a08142-tls-certs\") pod \"auth-test-model-c50e10-kserve-6df69ff995-f8gvd\" (UID: \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:39:46.212690 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.212647    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e66f5234-dc9b-4fc6-9d16-c520f1a08142-tmp-dir\") pod \"auth-test-model-c50e10-kserve-6df69ff995-f8gvd\" (UID: \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:39:46.212899 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.212754    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e66f5234-dc9b-4fc6-9d16-c520f1a08142-dshm\") pod \"auth-test-model-c50e10-kserve-6df69ff995-f8gvd\" (UID: \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:39:46.212899 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.212807    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9clwf\" (UniqueName: \"kubernetes.io/projected/e66f5234-dc9b-4fc6-9d16-c520f1a08142-kube-api-access-9clwf\") pod \"auth-test-model-c50e10-kserve-6df69ff995-f8gvd\" (UID: \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:39:46.313967 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.313923    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e66f5234-dc9b-4fc6-9d16-c520f1a08142-dshm\") pod \"auth-test-model-c50e10-kserve-6df69ff995-f8gvd\" (UID: \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:39:46.314174 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.313992    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9clwf\" (UniqueName: \"kubernetes.io/projected/e66f5234-dc9b-4fc6-9d16-c520f1a08142-kube-api-access-9clwf\") pod \"auth-test-model-c50e10-kserve-6df69ff995-f8gvd\" (UID: \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:39:46.314174 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.314038    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e66f5234-dc9b-4fc6-9d16-c520f1a08142-home\") pod \"auth-test-model-c50e10-kserve-6df69ff995-f8gvd\" (UID: \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:39:46.314174 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.314077    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66f5234-dc9b-4fc6-9d16-c520f1a08142-model-cache\") pod \"auth-test-model-c50e10-kserve-6df69ff995-f8gvd\" (UID: \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:39:46.314365 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.314167    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e66f5234-dc9b-4fc6-9d16-c520f1a08142-tls-certs\") pod \"auth-test-model-c50e10-kserve-6df69ff995-f8gvd\" (UID: \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:39:46.314365 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.314244    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e66f5234-dc9b-4fc6-9d16-c520f1a08142-tmp-dir\") pod \"auth-test-model-c50e10-kserve-6df69ff995-f8gvd\" (UID: \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:39:46.314507 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.314484    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e66f5234-dc9b-4fc6-9d16-c520f1a08142-home\") pod \"auth-test-model-c50e10-kserve-6df69ff995-f8gvd\" (UID: \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:39:46.314571 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.314508    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66f5234-dc9b-4fc6-9d16-c520f1a08142-model-cache\") pod \"auth-test-model-c50e10-kserve-6df69ff995-f8gvd\" (UID: \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:39:46.314650 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.314587    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e66f5234-dc9b-4fc6-9d16-c520f1a08142-tmp-dir\") pod \"auth-test-model-c50e10-kserve-6df69ff995-f8gvd\" (UID: \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:39:46.316383 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.316361    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e66f5234-dc9b-4fc6-9d16-c520f1a08142-dshm\") pod \"auth-test-model-c50e10-kserve-6df69ff995-f8gvd\" (UID: \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:39:46.316548 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.316532    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e66f5234-dc9b-4fc6-9d16-c520f1a08142-tls-certs\") pod \"auth-test-model-c50e10-kserve-6df69ff995-f8gvd\" (UID: \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:39:46.323346 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.323324    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9clwf\" (UniqueName: \"kubernetes.io/projected/e66f5234-dc9b-4fc6-9d16-c520f1a08142-kube-api-access-9clwf\") pod \"auth-test-model-c50e10-kserve-6df69ff995-f8gvd\" (UID: \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:39:46.395971 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.395930    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:39:46.670272 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:46.670244    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"]
Sep 18 13:39:46.672252 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:39:46.672220    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode66f5234_dc9b_4fc6_9d16_c520f1a08142.slice/crio-e919f9a39bc9bad0d448de384b01429112a856a6d4f59f3e5451e202d8ff0f46 WatchSource:0}: Error finding container e919f9a39bc9bad0d448de384b01429112a856a6d4f59f3e5451e202d8ff0f46: Status 404 returned error can't find the container with id e919f9a39bc9bad0d448de384b01429112a856a6d4f59f3e5451e202d8ff0f46
Sep 18 13:39:47.192455 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:47.192411    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd" event={"ID":"e66f5234-dc9b-4fc6-9d16-c520f1a08142","Type":"ContainerStarted","Data":"dc534d2d58a80259e606e83b921497c3250d3ac7bc39c86a440d30b3abf6cf05"}
Sep 18 13:39:47.192455 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:47.192461    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd" event={"ID":"e66f5234-dc9b-4fc6-9d16-c520f1a08142","Type":"ContainerStarted","Data":"e919f9a39bc9bad0d448de384b01429112a856a6d4f59f3e5451e202d8ff0f46"}
Sep 18 13:39:47.215585 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:47.215533    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd" podStartSLOduration=1.215516592 podStartE2EDuration="1.215516592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:39:47.213848228 +0000 UTC m=+1495.648724634" watchObservedRunningTime="2026-09-18 13:39:47.215516592 +0000 UTC m=+1495.650392996"
Sep 18 13:39:52.098567 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:52.098530    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k86vn_55e33e14-3f7a-4793-a724-9fd5022740d7/console-operator/1.log"
Sep 18 13:39:52.103610 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:52.103569    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k86vn_55e33e14-3f7a-4793-a724-9fd5022740d7/console-operator/1.log"
Sep 18 13:39:56.397059 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:56.397022    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:39:56.397458 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:56.397304    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:39:56.409995 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:56.409962    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:39:57.262018 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:39:57.261980    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:40:45.570261 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:45.570227    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"]
Sep 18 13:40:45.570756 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:45.570462    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j" podUID="3e87c7b2-c903-4be2-9039-3499f78e2cbb" containerName="istio-proxy" containerID="cri-o://9a30be30d80af2fced517fcdd791cad6e976b0ff51c7d86fd00fed961e336ace" gracePeriod=30
Sep 18 13:40:49.200582 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:49.200544    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"]
Sep 18 13:40:49.204286 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:49.204269    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:40:49.206862 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:49.206843    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5be8a3\"/\"distinct-2-w5-5be8a3-kserve-self-signed-certs\""
Sep 18 13:40:49.219839 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:49.219810    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"]
Sep 18 13:40:49.347921 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:49.347880    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f3f049-86e4-40ba-959f-be09abb3138f-model-cache\") pod \"distinct-2-w5-5be8a3-kserve-75c7c795-msncj\" (UID: \"80f3f049-86e4-40ba-959f-be09abb3138f\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:40:49.347921 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:49.347922    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f3f049-86e4-40ba-959f-be09abb3138f-tmp-dir\") pod \"distinct-2-w5-5be8a3-kserve-75c7c795-msncj\" (UID: \"80f3f049-86e4-40ba-959f-be09abb3138f\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:40:49.348138 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:49.348038    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f3f049-86e4-40ba-959f-be09abb3138f-dshm\") pod \"distinct-2-w5-5be8a3-kserve-75c7c795-msncj\" (UID: \"80f3f049-86e4-40ba-959f-be09abb3138f\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:40:49.348138 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:49.348085    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f3f049-86e4-40ba-959f-be09abb3138f-tls-certs\") pod \"distinct-2-w5-5be8a3-kserve-75c7c795-msncj\" (UID: \"80f3f049-86e4-40ba-959f-be09abb3138f\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:40:49.348210 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:49.348179    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f3f049-86e4-40ba-959f-be09abb3138f-home\") pod \"distinct-2-w5-5be8a3-kserve-75c7c795-msncj\" (UID: \"80f3f049-86e4-40ba-959f-be09abb3138f\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:40:49.348245 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:49.348228    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm7vw\" (UniqueName: \"kubernetes.io/projected/80f3f049-86e4-40ba-959f-be09abb3138f-kube-api-access-cm7vw\") pod \"distinct-2-w5-5be8a3-kserve-75c7c795-msncj\" (UID: \"80f3f049-86e4-40ba-959f-be09abb3138f\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:40:49.449184 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:49.449141    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f3f049-86e4-40ba-959f-be09abb3138f-home\") pod \"distinct-2-w5-5be8a3-kserve-75c7c795-msncj\" (UID: \"80f3f049-86e4-40ba-959f-be09abb3138f\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:40:49.449184 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:49.449191    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm7vw\" (UniqueName: \"kubernetes.io/projected/80f3f049-86e4-40ba-959f-be09abb3138f-kube-api-access-cm7vw\") pod \"distinct-2-w5-5be8a3-kserve-75c7c795-msncj\" (UID: \"80f3f049-86e4-40ba-959f-be09abb3138f\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:40:49.449429 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:49.449224    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f3f049-86e4-40ba-959f-be09abb3138f-model-cache\") pod \"distinct-2-w5-5be8a3-kserve-75c7c795-msncj\" (UID: \"80f3f049-86e4-40ba-959f-be09abb3138f\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:40:49.449429 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:49.449242    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f3f049-86e4-40ba-959f-be09abb3138f-tmp-dir\") pod \"distinct-2-w5-5be8a3-kserve-75c7c795-msncj\" (UID: \"80f3f049-86e4-40ba-959f-be09abb3138f\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:40:49.449429 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:49.449287    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f3f049-86e4-40ba-959f-be09abb3138f-dshm\") pod \"distinct-2-w5-5be8a3-kserve-75c7c795-msncj\" (UID: \"80f3f049-86e4-40ba-959f-be09abb3138f\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:40:49.449429 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:49.449351    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f3f049-86e4-40ba-959f-be09abb3138f-tls-certs\") pod \"distinct-2-w5-5be8a3-kserve-75c7c795-msncj\" (UID: \"80f3f049-86e4-40ba-959f-be09abb3138f\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:40:49.449658 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:49.449572    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f3f049-86e4-40ba-959f-be09abb3138f-home\") pod \"distinct-2-w5-5be8a3-kserve-75c7c795-msncj\" (UID: \"80f3f049-86e4-40ba-959f-be09abb3138f\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:40:49.449658 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:49.449623    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f3f049-86e4-40ba-959f-be09abb3138f-tmp-dir\") pod \"distinct-2-w5-5be8a3-kserve-75c7c795-msncj\" (UID: \"80f3f049-86e4-40ba-959f-be09abb3138f\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:40:49.449776 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:49.449755    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f3f049-86e4-40ba-959f-be09abb3138f-model-cache\") pod \"distinct-2-w5-5be8a3-kserve-75c7c795-msncj\" (UID: \"80f3f049-86e4-40ba-959f-be09abb3138f\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:40:49.451640 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:49.451549    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f3f049-86e4-40ba-959f-be09abb3138f-dshm\") pod \"distinct-2-w5-5be8a3-kserve-75c7c795-msncj\" (UID: \"80f3f049-86e4-40ba-959f-be09abb3138f\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:40:49.452017 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:49.451999    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f3f049-86e4-40ba-959f-be09abb3138f-tls-certs\") pod \"distinct-2-w5-5be8a3-kserve-75c7c795-msncj\" (UID: \"80f3f049-86e4-40ba-959f-be09abb3138f\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:40:49.459623 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:49.459583    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm7vw\" (UniqueName: \"kubernetes.io/projected/80f3f049-86e4-40ba-959f-be09abb3138f-kube-api-access-cm7vw\") pod \"distinct-2-w5-5be8a3-kserve-75c7c795-msncj\" (UID: \"80f3f049-86e4-40ba-959f-be09abb3138f\") " pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:40:49.514186 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:49.514153    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:40:49.804031 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:49.804005    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"]
Sep 18 13:40:49.806814 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:40:49.806782    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80f3f049_86e4_40ba_959f_be09abb3138f.slice/crio-7d0c75a7a531f9fdb6bf8dbb6e454a15121f4c5323081fd1ed051e3d4b65826e WatchSource:0}: Error finding container 7d0c75a7a531f9fdb6bf8dbb6e454a15121f4c5323081fd1ed051e3d4b65826e: Status 404 returned error can't find the container with id 7d0c75a7a531f9fdb6bf8dbb6e454a15121f4c5323081fd1ed051e3d4b65826e
Sep 18 13:40:50.242591 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.242550    2566 patch_prober.go:28] interesting pod/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 13:40:50.242992 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.242636    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j" podUID="3e87c7b2-c903-4be2-9039-3499f78e2cbb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 13:40:50.557178 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.557091    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj" event={"ID":"80f3f049-86e4-40ba-959f-be09abb3138f","Type":"ContainerStarted","Data":"136eaec3eb4bc684b87bd71cb35707515ac81548cd90e03c5e5cdf4226bc13e7"}
Sep 18 13:40:50.557178 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.557133    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj" event={"ID":"80f3f049-86e4-40ba-959f-be09abb3138f","Type":"ContainerStarted","Data":"7d0c75a7a531f9fdb6bf8dbb6e454a15121f4c5323081fd1ed051e3d4b65826e"}
Sep 18 13:40:50.577820 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.577761    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj" podStartSLOduration=1.577745768 podStartE2EDuration="1.577745768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:40:50.575316668 +0000 UTC m=+1559.010193078" watchObservedRunningTime="2026-09-18 13:40:50.577745768 +0000 UTC m=+1559.012622172"
Sep 18 13:40:50.853909 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.853880    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:40:50.964494 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.964455    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jf725\" (UniqueName: \"kubernetes.io/projected/3e87c7b2-c903-4be2-9039-3499f78e2cbb-kube-api-access-jf725\") pod \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") "
Sep 18 13:40:50.964696 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.964553    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istio-podinfo\") pod \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") "
Sep 18 13:40:50.964696 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.964589    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istio-data\") pod \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") "
Sep 18 13:40:50.964696 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.964656    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-credential-socket\") pod \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") "
Sep 18 13:40:50.964696 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.964680    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-workload-certs\") pod \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") "
Sep 18 13:40:50.964917 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.964718    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istio-token\") pod \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") "
Sep 18 13:40:50.964917 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.964793    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-workload-socket\") pod \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") "
Sep 18 13:40:50.964917 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.964886    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istiod-ca-cert\") pod \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") "
Sep 18 13:40:50.964917 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.964910    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istio-envoy\") pod \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\" (UID: \"3e87c7b2-c903-4be2-9039-3499f78e2cbb\") "
Sep 18 13:40:50.965127 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.964983    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istio-data" (OuterVolumeSpecName: "istio-data") pod "3e87c7b2-c903-4be2-9039-3499f78e2cbb" (UID: "3e87c7b2-c903-4be2-9039-3499f78e2cbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:40:50.965254 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.965221    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3e87c7b2-c903-4be2-9039-3499f78e2cbb" (UID: "3e87c7b2-c903-4be2-9039-3499f78e2cbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:40:50.965614 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.965363    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3e87c7b2-c903-4be2-9039-3499f78e2cbb" (UID: "3e87c7b2-c903-4be2-9039-3499f78e2cbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:40:50.965614 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.965378    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3e87c7b2-c903-4be2-9039-3499f78e2cbb" (UID: "3e87c7b2-c903-4be2-9039-3499f78e2cbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:40:50.965614 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.965436    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3e87c7b2-c903-4be2-9039-3499f78e2cbb" (UID: "3e87c7b2-c903-4be2-9039-3499f78e2cbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:40:50.965902 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.965862    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istio-data\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:40:50.966072 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.966050    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-credential-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:40:50.966072 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.966072    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-workload-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:40:50.966230 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.966088    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-workload-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:40:50.966230 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.966100    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istiod-ca-cert\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:40:50.967302 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.967274    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3e87c7b2-c903-4be2-9039-3499f78e2cbb" (UID: "3e87c7b2-c903-4be2-9039-3499f78e2cbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 13:40:50.967302 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.967294    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e87c7b2-c903-4be2-9039-3499f78e2cbb-kube-api-access-jf725" (OuterVolumeSpecName: "kube-api-access-jf725") pod "3e87c7b2-c903-4be2-9039-3499f78e2cbb" (UID: "3e87c7b2-c903-4be2-9039-3499f78e2cbb"). InnerVolumeSpecName "kube-api-access-jf725". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:40:50.967458 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.967405    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istio-token" (OuterVolumeSpecName: "istio-token") pod "3e87c7b2-c903-4be2-9039-3499f78e2cbb" (UID: "3e87c7b2-c903-4be2-9039-3499f78e2cbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:40:50.967708 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:50.967693    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3e87c7b2-c903-4be2-9039-3499f78e2cbb" (UID: "3e87c7b2-c903-4be2-9039-3499f78e2cbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:40:51.067681 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:51.067646    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istio-envoy\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:40:51.067681 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:51.067675    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf725\" (UniqueName: \"kubernetes.io/projected/3e87c7b2-c903-4be2-9039-3499f78e2cbb-kube-api-access-jf725\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:40:51.067681 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:51.067684    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istio-podinfo\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:40:51.067927 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:51.067693    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e87c7b2-c903-4be2-9039-3499f78e2cbb-istio-token\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:40:51.564336 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:51.564298    2566 generic.go:356] "Generic (PLEG): container finished" podID="3e87c7b2-c903-4be2-9039-3499f78e2cbb" containerID="9a30be30d80af2fced517fcdd791cad6e976b0ff51c7d86fd00fed961e336ace" exitCode=0
Sep 18 13:40:51.564781 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:51.564342    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"
Sep 18 13:40:51.564781 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:51.564384    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j" event={"ID":"3e87c7b2-c903-4be2-9039-3499f78e2cbb","Type":"ContainerDied","Data":"9a30be30d80af2fced517fcdd791cad6e976b0ff51c7d86fd00fed961e336ace"}
Sep 18 13:40:51.564781 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:51.564435    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j" event={"ID":"3e87c7b2-c903-4be2-9039-3499f78e2cbb","Type":"ContainerDied","Data":"d932f65bc35e33056cf106d475ff83ec462d0554d6f32035fd75a7dcd38409c4"}
Sep 18 13:40:51.564781 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:51.564463    2566 scope.go:117] "RemoveContainer" containerID="9a30be30d80af2fced517fcdd791cad6e976b0ff51c7d86fd00fed961e336ace"
Sep 18 13:40:51.574260 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:51.574242    2566 scope.go:117] "RemoveContainer" containerID="f1659a0bde9830086be0299d3bc16d4c61a28de557c91418319db5a7a3bcd590"
Sep 18 13:40:51.582929 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:51.582908    2566 scope.go:117] "RemoveContainer" containerID="9a30be30d80af2fced517fcdd791cad6e976b0ff51c7d86fd00fed961e336ace"
Sep 18 13:40:51.583193 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:40:51.583172    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a30be30d80af2fced517fcdd791cad6e976b0ff51c7d86fd00fed961e336ace\": container with ID starting with 9a30be30d80af2fced517fcdd791cad6e976b0ff51c7d86fd00fed961e336ace not found: ID does not exist" containerID="9a30be30d80af2fced517fcdd791cad6e976b0ff51c7d86fd00fed961e336ace"
Sep 18 13:40:51.583274 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:51.583204    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a30be30d80af2fced517fcdd791cad6e976b0ff51c7d86fd00fed961e336ace"} err="failed to get container status \"9a30be30d80af2fced517fcdd791cad6e976b0ff51c7d86fd00fed961e336ace\": rpc error: code = NotFound desc = could not find container \"9a30be30d80af2fced517fcdd791cad6e976b0ff51c7d86fd00fed961e336ace\": container with ID starting with 9a30be30d80af2fced517fcdd791cad6e976b0ff51c7d86fd00fed961e336ace not found: ID does not exist"
Sep 18 13:40:51.583274 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:51.583227    2566 scope.go:117] "RemoveContainer" containerID="f1659a0bde9830086be0299d3bc16d4c61a28de557c91418319db5a7a3bcd590"
Sep 18 13:40:51.583507 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:40:51.583488    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1659a0bde9830086be0299d3bc16d4c61a28de557c91418319db5a7a3bcd590\": container with ID starting with f1659a0bde9830086be0299d3bc16d4c61a28de557c91418319db5a7a3bcd590 not found: ID does not exist" containerID="f1659a0bde9830086be0299d3bc16d4c61a28de557c91418319db5a7a3bcd590"
Sep 18 13:40:51.583552 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:51.583513    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1659a0bde9830086be0299d3bc16d4c61a28de557c91418319db5a7a3bcd590"} err="failed to get container status \"f1659a0bde9830086be0299d3bc16d4c61a28de557c91418319db5a7a3bcd590\": rpc error: code = NotFound desc = could not find container \"f1659a0bde9830086be0299d3bc16d4c61a28de557c91418319db5a7a3bcd590\": container with ID starting with f1659a0bde9830086be0299d3bc16d4c61a28de557c91418319db5a7a3bcd590 not found: ID does not exist"
Sep 18 13:40:51.588058 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:51.588030    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"]
Sep 18 13:40:51.593607 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:51.593570    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fa9905fd-openshift-default-66587b7cc6-bbq4j"]
Sep 18 13:40:52.078263 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:52.078219    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e87c7b2-c903-4be2-9039-3499f78e2cbb" path="/var/lib/kubelet/pods/3e87c7b2-c903-4be2-9039-3499f78e2cbb/volumes"
Sep 18 13:40:52.310829 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:52.310791    2566 scope.go:117] "RemoveContainer" containerID="12211cda556a51ed03eb5b8d900fc67b0e8018b9e34cbefd128140b4a78bc2a8"
Sep 18 13:40:52.319546 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:52.319518    2566 scope.go:117] "RemoveContainer" containerID="bcef7297356f46ba3106bd00d6a5e9cacb0f19593d21a938b18437ffdf5ee2a7"
Sep 18 13:40:59.514435 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:59.514394    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:40:59.514850 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:59.514662    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:40:59.526967 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:59.526946    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:40:59.617108 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:40:59.617074    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:41:06.875957 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:06.875922    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"]
Sep 18 13:41:06.876363 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:06.876340    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e87c7b2-c903-4be2-9039-3499f78e2cbb" containerName="istio-proxy"
Sep 18 13:41:06.876363 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:06.876351    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e87c7b2-c903-4be2-9039-3499f78e2cbb" containerName="istio-proxy"
Sep 18 13:41:06.876437 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:06.876377    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e87c7b2-c903-4be2-9039-3499f78e2cbb" containerName="istio-proxy"
Sep 18 13:41:06.876437 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:06.876383    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e87c7b2-c903-4be2-9039-3499f78e2cbb" containerName="istio-proxy"
Sep 18 13:41:06.876503 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:06.876454    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e87c7b2-c903-4be2-9039-3499f78e2cbb" containerName="istio-proxy"
Sep 18 13:41:06.876503 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:06.876466    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e87c7b2-c903-4be2-9039-3499f78e2cbb" containerName="istio-proxy"
Sep 18 13:41:06.879570 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:06.879547    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:41:06.882022 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:06.881997    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-951469\"/\"embedding-w0-951469-kserve-self-signed-certs\""
Sep 18 13:41:06.888608 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:06.888565    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"]
Sep 18 13:41:06.936726 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:06.936695    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ad330a-f565-4baf-827e-535cb5864563-home\") pod \"embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf\" (UID: \"d6ad330a-f565-4baf-827e-535cb5864563\") " pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:41:06.936898 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:06.936756    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ad330a-f565-4baf-827e-535cb5864563-model-cache\") pod \"embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf\" (UID: \"d6ad330a-f565-4baf-827e-535cb5864563\") " pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:41:06.936898 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:06.936799    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ad330a-f565-4baf-827e-535cb5864563-tmp-dir\") pod \"embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf\" (UID: \"d6ad330a-f565-4baf-827e-535cb5864563\") " pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:41:06.936898 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:06.936836    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ad330a-f565-4baf-827e-535cb5864563-dshm\") pod \"embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf\" (UID: \"d6ad330a-f565-4baf-827e-535cb5864563\") " pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:41:06.936898 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:06.936858    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ad330a-f565-4baf-827e-535cb5864563-tls-certs\") pod \"embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf\" (UID: \"d6ad330a-f565-4baf-827e-535cb5864563\") " pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:41:06.936898 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:06.936875    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjwgv\" (UniqueName: \"kubernetes.io/projected/d6ad330a-f565-4baf-827e-535cb5864563-kube-api-access-fjwgv\") pod \"embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf\" (UID: \"d6ad330a-f565-4baf-827e-535cb5864563\") " pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:41:07.044969 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:07.044923    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ad330a-f565-4baf-827e-535cb5864563-home\") pod \"embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf\" (UID: \"d6ad330a-f565-4baf-827e-535cb5864563\") " pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:41:07.045165 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:07.045082    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ad330a-f565-4baf-827e-535cb5864563-model-cache\") pod \"embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf\" (UID: \"d6ad330a-f565-4baf-827e-535cb5864563\") " pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:41:07.045165 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:07.045130    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ad330a-f565-4baf-827e-535cb5864563-tmp-dir\") pod \"embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf\" (UID: \"d6ad330a-f565-4baf-827e-535cb5864563\") " pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:41:07.045295 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:07.045207    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ad330a-f565-4baf-827e-535cb5864563-dshm\") pod \"embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf\" (UID: \"d6ad330a-f565-4baf-827e-535cb5864563\") " pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:41:07.045295 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:07.045243    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ad330a-f565-4baf-827e-535cb5864563-tls-certs\") pod \"embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf\" (UID: \"d6ad330a-f565-4baf-827e-535cb5864563\") " pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:41:07.045295 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:07.045274    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjwgv\" (UniqueName: \"kubernetes.io/projected/d6ad330a-f565-4baf-827e-535cb5864563-kube-api-access-fjwgv\") pod \"embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf\" (UID: \"d6ad330a-f565-4baf-827e-535cb5864563\") " pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:41:07.045943 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:07.045912    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ad330a-f565-4baf-827e-535cb5864563-tmp-dir\") pod \"embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf\" (UID: \"d6ad330a-f565-4baf-827e-535cb5864563\") " pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:41:07.046506 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:07.046452    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ad330a-f565-4baf-827e-535cb5864563-model-cache\") pod \"embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf\" (UID: \"d6ad330a-f565-4baf-827e-535cb5864563\") " pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:41:07.046792 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:07.046770    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ad330a-f565-4baf-827e-535cb5864563-home\") pod \"embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf\" (UID: \"d6ad330a-f565-4baf-827e-535cb5864563\") " pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:41:07.049167 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:07.049141    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ad330a-f565-4baf-827e-535cb5864563-tls-certs\") pod \"embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf\" (UID: \"d6ad330a-f565-4baf-827e-535cb5864563\") " pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:41:07.049800 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:07.049777    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ad330a-f565-4baf-827e-535cb5864563-dshm\") pod \"embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf\" (UID: \"d6ad330a-f565-4baf-827e-535cb5864563\") " pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:41:07.055802 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:07.055778    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjwgv\" (UniqueName: \"kubernetes.io/projected/d6ad330a-f565-4baf-827e-535cb5864563-kube-api-access-fjwgv\") pod \"embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf\" (UID: \"d6ad330a-f565-4baf-827e-535cb5864563\") " pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:41:07.189678 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:07.189562    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:41:07.469455 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:07.469422    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"]
Sep 18 13:41:07.471243 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:41:07.471214    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6ad330a_f565_4baf_827e_535cb5864563.slice/crio-13befb80734a041456fdecdefb945b7d5cba96445c5fe7ccff96728a80186869 WatchSource:0}: Error finding container 13befb80734a041456fdecdefb945b7d5cba96445c5fe7ccff96728a80186869: Status 404 returned error can't find the container with id 13befb80734a041456fdecdefb945b7d5cba96445c5fe7ccff96728a80186869
Sep 18 13:41:07.650918 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:07.650877    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf" event={"ID":"d6ad330a-f565-4baf-827e-535cb5864563","Type":"ContainerStarted","Data":"e8297b776afd9ef04ef031727bee2ac2692d908bb2da8f598aef641b509c0c33"}
Sep 18 13:41:07.650918 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:07.650926    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf" event={"ID":"d6ad330a-f565-4baf-827e-535cb5864563","Type":"ContainerStarted","Data":"13befb80734a041456fdecdefb945b7d5cba96445c5fe7ccff96728a80186869"}
Sep 18 13:41:07.668705 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:07.668654    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf" podStartSLOduration=1.6686383980000001 podStartE2EDuration="1.668638398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:41:07.667244357 +0000 UTC m=+1576.102120761" watchObservedRunningTime="2026-09-18 13:41:07.668638398 +0000 UTC m=+1576.103514849"
Sep 18 13:41:17.190705 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:17.190664    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:41:17.191173 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:17.190720    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:41:17.203266 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:17.203233    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:41:17.718893 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:17.718854    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:41:52.782203 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:52.782117    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk"]
Sep 18 13:41:52.785719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:52.785703    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk"
Sep 18 13:41:52.788656 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:52.788633    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-bc95f3-dockercfg-zdn65\""
Sep 18 13:41:52.789079 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:52.789060    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-bc95f3\""
Sep 18 13:41:52.801002 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:52.800978    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk"]
Sep 18 13:41:52.937477 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:52.937443    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gnjj\" (UniqueName: \"kubernetes.io/projected/b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40-kube-api-access-5gnjj\") pod \"payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk\" (UID: \"b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk"
Sep 18 13:41:52.937691 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:52.937487    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk\" (UID: \"b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk"
Sep 18 13:41:53.039002 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:53.038911    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gnjj\" (UniqueName: \"kubernetes.io/projected/b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40-kube-api-access-5gnjj\") pod \"payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk\" (UID: \"b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk"
Sep 18 13:41:53.039002 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:53.038956    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk\" (UID: \"b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk"
Sep 18 13:41:53.039688 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:53.039668    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk\" (UID: \"b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk"
Sep 18 13:41:53.048112 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:53.048088    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gnjj\" (UniqueName: \"kubernetes.io/projected/b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40-kube-api-access-5gnjj\") pod \"payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk\" (UID: \"b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk"
Sep 18 13:41:53.096261 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:53.096228    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk"
Sep 18 13:41:53.371960 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:53.371933    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk"]
Sep 18 13:41:53.374009 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:41:53.373966    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb84a87a4_ca6b_4f24_ac8c_8b8bd2928a40.slice/crio-686e6ee269270eed52c6ece76258437bee710495638962cdd51ddb22b751dec3 WatchSource:0}: Error finding container 686e6ee269270eed52c6ece76258437bee710495638962cdd51ddb22b751dec3: Status 404 returned error can't find the container with id 686e6ee269270eed52c6ece76258437bee710495638962cdd51ddb22b751dec3
Sep 18 13:41:53.910776 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:53.910741    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk" event={"ID":"b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40","Type":"ContainerStarted","Data":"bcfcebc68aac3c55b7dfe53546ebb2edbcb5d67745f8964a7ebeeebbe1279686"}
Sep 18 13:41:53.911205 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:53.910795    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk" event={"ID":"b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40","Type":"ContainerStarted","Data":"686e6ee269270eed52c6ece76258437bee710495638962cdd51ddb22b751dec3"}
Sep 18 13:41:53.911205 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:53.910827    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk"
Sep 18 13:41:53.931126 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:41:53.931076    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk" podStartSLOduration=1.931061307 podStartE2EDuration="1.931061307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:41:53.930229229 +0000 UTC m=+1622.365105634" watchObservedRunningTime="2026-09-18 13:41:53.931061307 +0000 UTC m=+1622.365937711"
Sep 18 13:42:04.918454 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:04.918420    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk"
Sep 18 13:42:22.127588 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:22.127550    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm"]
Sep 18 13:42:22.131339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:22.131317    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm"
Sep 18 13:42:22.133671 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:22.133644    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-57732b\""
Sep 18 13:42:22.133779 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:22.133679    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-57732b-dockercfg-qhc2q\""
Sep 18 13:42:22.143723 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:22.143698    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm"]
Sep 18 13:42:22.183679 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:22.183644    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h"]
Sep 18 13:42:22.187269 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:22.187246    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h"
Sep 18 13:42:22.214076 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:22.214043    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h"]
Sep 18 13:42:22.225811 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:22.225776    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/face956b-dd2e-4bd3-8a43-a5c9f875f25e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm\" (UID: \"face956b-dd2e-4bd3-8a43-a5c9f875f25e\") " pod="openshift-ingress/payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm"
Sep 18 13:42:22.225995 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:22.225924    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q47b\" (UniqueName: \"kubernetes.io/projected/face956b-dd2e-4bd3-8a43-a5c9f875f25e-kube-api-access-7q47b\") pod \"payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm\" (UID: \"face956b-dd2e-4bd3-8a43-a5c9f875f25e\") " pod="openshift-ingress/payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm"
Sep 18 13:42:22.326707 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:22.326579    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/face956b-dd2e-4bd3-8a43-a5c9f875f25e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm\" (UID: \"face956b-dd2e-4bd3-8a43-a5c9f875f25e\") " pod="openshift-ingress/payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm"
Sep 18 13:42:22.326884 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:22.326747    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q47b\" (UniqueName: \"kubernetes.io/projected/face956b-dd2e-4bd3-8a43-a5c9f875f25e-kube-api-access-7q47b\") pod \"payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm\" (UID: \"face956b-dd2e-4bd3-8a43-a5c9f875f25e\") " pod="openshift-ingress/payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm"
Sep 18 13:42:22.326884 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:22.326791    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnw9p\" (UniqueName: \"kubernetes.io/projected/9423c4c1-2af6-46d6-86e6-7df836abe290-kube-api-access-lnw9p\") pod \"payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h\" (UID: \"9423c4c1-2af6-46d6-86e6-7df836abe290\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h"
Sep 18 13:42:22.326980 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:22.326912    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9423c4c1-2af6-46d6-86e6-7df836abe290-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h\" (UID: \"9423c4c1-2af6-46d6-86e6-7df836abe290\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h"
Sep 18 13:42:22.327259 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:22.327237    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/face956b-dd2e-4bd3-8a43-a5c9f875f25e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm\" (UID: \"face956b-dd2e-4bd3-8a43-a5c9f875f25e\") " pod="openshift-ingress/payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm"
Sep 18 13:42:22.335540 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:22.335512    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q47b\" (UniqueName: \"kubernetes.io/projected/face956b-dd2e-4bd3-8a43-a5c9f875f25e-kube-api-access-7q47b\") pod \"payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm\" (UID: \"face956b-dd2e-4bd3-8a43-a5c9f875f25e\") " pod="openshift-ingress/payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm"
Sep 18 13:42:22.428175 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:22.428075    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnw9p\" (UniqueName: \"kubernetes.io/projected/9423c4c1-2af6-46d6-86e6-7df836abe290-kube-api-access-lnw9p\") pod \"payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h\" (UID: \"9423c4c1-2af6-46d6-86e6-7df836abe290\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h"
Sep 18 13:42:22.428175 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:22.428160    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9423c4c1-2af6-46d6-86e6-7df836abe290-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h\" (UID: \"9423c4c1-2af6-46d6-86e6-7df836abe290\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h"
Sep 18 13:42:22.428864 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:22.428845    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9423c4c1-2af6-46d6-86e6-7df836abe290-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h\" (UID: \"9423c4c1-2af6-46d6-86e6-7df836abe290\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h"
Sep 18 13:42:22.437044 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:22.437020    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnw9p\" (UniqueName: \"kubernetes.io/projected/9423c4c1-2af6-46d6-86e6-7df836abe290-kube-api-access-lnw9p\") pod \"payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h\" (UID: \"9423c4c1-2af6-46d6-86e6-7df836abe290\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h"
Sep 18 13:42:22.443717 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:22.443697    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm"
Sep 18 13:42:22.497200 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:22.497171    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h"
Sep 18 13:42:22.727341 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:22.727121    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm"]
Sep 18 13:42:22.730271 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:42:22.730242    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podface956b_dd2e_4bd3_8a43_a5c9f875f25e.slice/crio-8486eb9d57a5239de63d4e0315948dd25509dc4d9f5feac6b3b436bfde79216e WatchSource:0}: Error finding container 8486eb9d57a5239de63d4e0315948dd25509dc4d9f5feac6b3b436bfde79216e: Status 404 returned error can't find the container with id 8486eb9d57a5239de63d4e0315948dd25509dc4d9f5feac6b3b436bfde79216e
Sep 18 13:42:22.790269 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:22.790242    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h"]
Sep 18 13:42:22.791833 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:42:22.791808    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9423c4c1_2af6_46d6_86e6_7df836abe290.slice/crio-fbad7ab45a0a23b484b140a4f79cdb566411375ed5efd7ea73008004165caf25 WatchSource:0}: Error finding container fbad7ab45a0a23b484b140a4f79cdb566411375ed5efd7ea73008004165caf25: Status 404 returned error can't find the container with id fbad7ab45a0a23b484b140a4f79cdb566411375ed5efd7ea73008004165caf25
Sep 18 13:42:23.082054 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:23.081928    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h" event={"ID":"9423c4c1-2af6-46d6-86e6-7df836abe290","Type":"ContainerStarted","Data":"40669338e30fa19dbf07994c92349bcced44fa1af0053476c78b580f30f7187d"}
Sep 18 13:42:23.082241 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:23.082084    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h" event={"ID":"9423c4c1-2af6-46d6-86e6-7df836abe290","Type":"ContainerStarted","Data":"fbad7ab45a0a23b484b140a4f79cdb566411375ed5efd7ea73008004165caf25"}
Sep 18 13:42:23.082241 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:23.082153    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h"
Sep 18 13:42:23.084862 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:23.084832    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm" event={"ID":"face956b-dd2e-4bd3-8a43-a5c9f875f25e","Type":"ContainerStarted","Data":"247d4c1f6ccebe306744dfed1032ae093a90ae507435ab49c8ba80df32644843"}
Sep 18 13:42:23.085000 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:23.084892    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm" event={"ID":"face956b-dd2e-4bd3-8a43-a5c9f875f25e","Type":"ContainerStarted","Data":"8486eb9d57a5239de63d4e0315948dd25509dc4d9f5feac6b3b436bfde79216e"}
Sep 18 13:42:23.085202 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:23.085147    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm"
Sep 18 13:42:23.103828 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:23.103764    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h" podStartSLOduration=1.103743273 podStartE2EDuration="1.103743273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:42:23.10083961 +0000 UTC m=+1651.535716013" watchObservedRunningTime="2026-09-18 13:42:23.103743273 +0000 UTC m=+1651.538619677"
Sep 18 13:42:23.124463 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:23.124395    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm" podStartSLOduration=1.124348492 podStartE2EDuration="1.124348492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:42:23.122285552 +0000 UTC m=+1651.557161956" watchObservedRunningTime="2026-09-18 13:42:23.124348492 +0000 UTC m=+1651.559224897"
Sep 18 13:42:34.092558 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:34.092523    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h"
Sep 18 13:42:34.093080 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:34.092979    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm"
Sep 18 13:42:48.013144 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:48.013105    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc95f3-qfkcc"]
Sep 18 13:42:48.017263 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:48.017243    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc95f3-qfkcc"
Sep 18 13:42:48.019502 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:48.019471    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:42:48.019805 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:48.019788    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nh2cd\""
Sep 18 13:42:48.031144 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:48.031120    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc95f3-qfkcc"]
Sep 18 13:42:48.098316 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:48.098277    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c62fb1-9b10-434b-a9a9-bb15de4958e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bc95f3-qfkcc\" (UID: \"02c62fb1-9b10-434b-a9a9-bb15de4958e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc95f3-qfkcc"
Sep 18 13:42:48.199689 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:48.199650    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c62fb1-9b10-434b-a9a9-bb15de4958e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bc95f3-qfkcc\" (UID: \"02c62fb1-9b10-434b-a9a9-bb15de4958e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc95f3-qfkcc"
Sep 18 13:42:48.200304 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:48.200280    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c62fb1-9b10-434b-a9a9-bb15de4958e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bc95f3-qfkcc\" (UID: \"02c62fb1-9b10-434b-a9a9-bb15de4958e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc95f3-qfkcc"
Sep 18 13:42:48.328362 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:48.328267    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc95f3-qfkcc"
Sep 18 13:42:48.604306 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:48.604271    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc95f3-qfkcc"]
Sep 18 13:42:48.607486 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:42:48.607461    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c62fb1_9b10_434b_a9a9_bb15de4958e7.slice/crio-c829d68cf123e90242c2170fa380a8713e779e537047a937343ce99231a5758b WatchSource:0}: Error finding container c829d68cf123e90242c2170fa380a8713e779e537047a937343ce99231a5758b: Status 404 returned error can't find the container with id c829d68cf123e90242c2170fa380a8713e779e537047a937343ce99231a5758b
Sep 18 13:42:49.247895 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:49.247799    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc95f3-qfkcc" event={"ID":"02c62fb1-9b10-434b-a9a9-bb15de4958e7","Type":"ContainerStarted","Data":"5a341bf05713ccb27a4fa9141b912974a66b4e09c5ac69c03fceb56ebcd4aa29"}
Sep 18 13:42:49.247895 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:49.247864    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc95f3-qfkcc" event={"ID":"02c62fb1-9b10-434b-a9a9-bb15de4958e7","Type":"ContainerStarted","Data":"c829d68cf123e90242c2170fa380a8713e779e537047a937343ce99231a5758b"}
Sep 18 13:42:49.314171 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:49.314120    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc95f3-qfkcc" podStartSLOduration=2.314105002 podStartE2EDuration="2.314105002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:42:49.309398795 +0000 UTC m=+1677.744275198" watchObservedRunningTime="2026-09-18 13:42:49.314105002 +0000 UTC m=+1677.748981457"
Sep 18 13:42:50.256200 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:50.256166    2566 generic.go:356] "Generic (PLEG): container finished" podID="02c62fb1-9b10-434b-a9a9-bb15de4958e7" containerID="5a341bf05713ccb27a4fa9141b912974a66b4e09c5ac69c03fceb56ebcd4aa29" exitCode=0
Sep 18 13:42:50.256664 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:50.256258    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc95f3-qfkcc" event={"ID":"02c62fb1-9b10-434b-a9a9-bb15de4958e7","Type":"ContainerDied","Data":"5a341bf05713ccb27a4fa9141b912974a66b4e09c5ac69c03fceb56ebcd4aa29"}
Sep 18 13:42:51.410783 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:51.410756    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc95f3-qfkcc"
Sep 18 13:42:51.533132 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:51.533036    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c62fb1-9b10-434b-a9a9-bb15de4958e7-maas-api-service-ca\") pod \"02c62fb1-9b10-434b-a9a9-bb15de4958e7\" (UID: \"02c62fb1-9b10-434b-a9a9-bb15de4958e7\") "
Sep 18 13:42:51.533405 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:51.533381    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c62fb1-9b10-434b-a9a9-bb15de4958e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "02c62fb1-9b10-434b-a9a9-bb15de4958e7" (UID: "02c62fb1-9b10-434b-a9a9-bb15de4958e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:42:51.634557 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:51.634511    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c62fb1-9b10-434b-a9a9-bb15de4958e7-maas-api-service-ca\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:42:52.270000 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:52.269963    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc95f3-qfkcc"
Sep 18 13:42:52.270168 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:52.270004    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc95f3-qfkcc" event={"ID":"02c62fb1-9b10-434b-a9a9-bb15de4958e7","Type":"ContainerDied","Data":"c829d68cf123e90242c2170fa380a8713e779e537047a937343ce99231a5758b"}
Sep 18 13:42:52.270168 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:52.270051    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c829d68cf123e90242c2170fa380a8713e779e537047a937343ce99231a5758b"
Sep 18 13:42:52.356432 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:42:52.356400    2566 scope.go:117] "RemoveContainer" containerID="e1fc196a542219706bd93725d7e842ed3e34b143b2472f9a82bebb5a9f5d13b3"
Sep 18 13:43:13.902682 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:13.902648    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk"]
Sep 18 13:43:13.903267 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:13.902870    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk" podUID="b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40" containerName="payload-pre-processing" containerID="cri-o://bcfcebc68aac3c55b7dfe53546ebb2edbcb5d67745f8964a7ebeeebbe1279686" gracePeriod=30
Sep 18 13:43:14.183318 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:14.183284    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc95f3-qfkcc"]
Sep 18 13:43:14.189199 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:14.189174    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk"
Sep 18 13:43:14.190405 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:14.190384    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bc95f3-qfkcc"]
Sep 18 13:43:14.268924 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:14.268895    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gnjj\" (UniqueName: \"kubernetes.io/projected/b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40-kube-api-access-5gnjj\") pod \"b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40\" (UID: \"b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40\") "
Sep 18 13:43:14.269089 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:14.268935    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40-plugins-config-volume\") pod \"b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40\" (UID: \"b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40\") "
Sep 18 13:43:14.269295 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:14.269265    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40" (UID: "b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:43:14.271104 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:14.271080    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40-kube-api-access-5gnjj" (OuterVolumeSpecName: "kube-api-access-5gnjj") pod "b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40" (UID: "b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40"). InnerVolumeSpecName "kube-api-access-5gnjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:43:14.369997 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:14.369966    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gnjj\" (UniqueName: \"kubernetes.io/projected/b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40-kube-api-access-5gnjj\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:43:14.369997 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:14.369994    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40-plugins-config-volume\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:43:14.408368 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:14.408335    2566 generic.go:356] "Generic (PLEG): container finished" podID="b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40" containerID="bcfcebc68aac3c55b7dfe53546ebb2edbcb5d67745f8964a7ebeeebbe1279686" exitCode=0
Sep 18 13:43:14.408510 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:14.408381    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk"
Sep 18 13:43:14.408510 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:14.408421    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk" event={"ID":"b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40","Type":"ContainerDied","Data":"bcfcebc68aac3c55b7dfe53546ebb2edbcb5d67745f8964a7ebeeebbe1279686"}
Sep 18 13:43:14.408510 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:14.408469    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk" event={"ID":"b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40","Type":"ContainerDied","Data":"686e6ee269270eed52c6ece76258437bee710495638962cdd51ddb22b751dec3"}
Sep 18 13:43:14.408510 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:14.408491    2566 scope.go:117] "RemoveContainer" containerID="bcfcebc68aac3c55b7dfe53546ebb2edbcb5d67745f8964a7ebeeebbe1279686"
Sep 18 13:43:14.423382 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:14.423365    2566 scope.go:117] "RemoveContainer" containerID="bcfcebc68aac3c55b7dfe53546ebb2edbcb5d67745f8964a7ebeeebbe1279686"
Sep 18 13:43:14.423683 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:43:14.423663    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcfcebc68aac3c55b7dfe53546ebb2edbcb5d67745f8964a7ebeeebbe1279686\": container with ID starting with bcfcebc68aac3c55b7dfe53546ebb2edbcb5d67745f8964a7ebeeebbe1279686 not found: ID does not exist" containerID="bcfcebc68aac3c55b7dfe53546ebb2edbcb5d67745f8964a7ebeeebbe1279686"
Sep 18 13:43:14.423747 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:14.423691    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcfcebc68aac3c55b7dfe53546ebb2edbcb5d67745f8964a7ebeeebbe1279686"} err="failed to get container status \"bcfcebc68aac3c55b7dfe53546ebb2edbcb5d67745f8964a7ebeeebbe1279686\": rpc error: code = NotFound desc = could not find container \"bcfcebc68aac3c55b7dfe53546ebb2edbcb5d67745f8964a7ebeeebbe1279686\": container with ID starting with bcfcebc68aac3c55b7dfe53546ebb2edbcb5d67745f8964a7ebeeebbe1279686 not found: ID does not exist"
Sep 18 13:43:14.446161 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:14.446129    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk"]
Sep 18 13:43:14.460583 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:14.460559    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bc95f3-794b866b6-r8pwk"]
Sep 18 13:43:16.078725 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:16.078690    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02c62fb1-9b10-434b-a9a9-bb15de4958e7" path="/var/lib/kubelet/pods/02c62fb1-9b10-434b-a9a9-bb15de4958e7/volumes"
Sep 18 13:43:16.079113 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:16.079099    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40" path="/var/lib/kubelet/pods/b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40/volumes"
Sep 18 13:43:47.408853 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:47.408819    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm"]
Sep 18 13:43:47.409452 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:47.409042    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm" podUID="face956b-dd2e-4bd3-8a43-a5c9f875f25e" containerName="payload-processing" containerID="cri-o://247d4c1f6ccebe306744dfed1032ae093a90ae507435ab49c8ba80df32644843" gracePeriod=30
Sep 18 13:43:47.411590 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:47.411562    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h"]
Sep 18 13:43:47.411837 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:47.411813    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h" podUID="9423c4c1-2af6-46d6-86e6-7df836abe290" containerName="payload-pre-processing" containerID="cri-o://40669338e30fa19dbf07994c92349bcced44fa1af0053476c78b580f30f7187d" gracePeriod=30
Sep 18 13:43:47.595907 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:47.595863    2566 generic.go:356] "Generic (PLEG): container finished" podID="face956b-dd2e-4bd3-8a43-a5c9f875f25e" containerID="247d4c1f6ccebe306744dfed1032ae093a90ae507435ab49c8ba80df32644843" exitCode=0
Sep 18 13:43:47.596066 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:47.596049    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm" event={"ID":"face956b-dd2e-4bd3-8a43-a5c9f875f25e","Type":"ContainerDied","Data":"247d4c1f6ccebe306744dfed1032ae093a90ae507435ab49c8ba80df32644843"}
Sep 18 13:43:47.598111 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:47.598084    2566 generic.go:356] "Generic (PLEG): container finished" podID="9423c4c1-2af6-46d6-86e6-7df836abe290" containerID="40669338e30fa19dbf07994c92349bcced44fa1af0053476c78b580f30f7187d" exitCode=0
Sep 18 13:43:47.598321 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:47.598262    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h" event={"ID":"9423c4c1-2af6-46d6-86e6-7df836abe290","Type":"ContainerDied","Data":"40669338e30fa19dbf07994c92349bcced44fa1af0053476c78b580f30f7187d"}
Sep 18 13:43:47.721537 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:47.721512    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm"
Sep 18 13:43:47.725220 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:47.725197    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h"
Sep 18 13:43:47.819777 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:47.819463    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9423c4c1-2af6-46d6-86e6-7df836abe290-plugins-config-volume\") pod \"9423c4c1-2af6-46d6-86e6-7df836abe290\" (UID: \"9423c4c1-2af6-46d6-86e6-7df836abe290\") "
Sep 18 13:43:47.819777 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:47.819564    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnw9p\" (UniqueName: \"kubernetes.io/projected/9423c4c1-2af6-46d6-86e6-7df836abe290-kube-api-access-lnw9p\") pod \"9423c4c1-2af6-46d6-86e6-7df836abe290\" (UID: \"9423c4c1-2af6-46d6-86e6-7df836abe290\") "
Sep 18 13:43:47.819777 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:47.819635    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/face956b-dd2e-4bd3-8a43-a5c9f875f25e-plugins-config-volume\") pod \"face956b-dd2e-4bd3-8a43-a5c9f875f25e\" (UID: \"face956b-dd2e-4bd3-8a43-a5c9f875f25e\") "
Sep 18 13:43:47.820104 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:47.819944    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7q47b\" (UniqueName: \"kubernetes.io/projected/face956b-dd2e-4bd3-8a43-a5c9f875f25e-kube-api-access-7q47b\") pod \"face956b-dd2e-4bd3-8a43-a5c9f875f25e\" (UID: \"face956b-dd2e-4bd3-8a43-a5c9f875f25e\") "
Sep 18 13:43:47.821350 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:47.821046    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/face956b-dd2e-4bd3-8a43-a5c9f875f25e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "face956b-dd2e-4bd3-8a43-a5c9f875f25e" (UID: "face956b-dd2e-4bd3-8a43-a5c9f875f25e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:43:47.822914 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:47.821711    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9423c4c1-2af6-46d6-86e6-7df836abe290-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9423c4c1-2af6-46d6-86e6-7df836abe290" (UID: "9423c4c1-2af6-46d6-86e6-7df836abe290"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:43:47.823749 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:47.823707    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/face956b-dd2e-4bd3-8a43-a5c9f875f25e-kube-api-access-7q47b" (OuterVolumeSpecName: "kube-api-access-7q47b") pod "face956b-dd2e-4bd3-8a43-a5c9f875f25e" (UID: "face956b-dd2e-4bd3-8a43-a5c9f875f25e"). InnerVolumeSpecName "kube-api-access-7q47b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:43:47.825062 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:47.825014    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9423c4c1-2af6-46d6-86e6-7df836abe290-kube-api-access-lnw9p" (OuterVolumeSpecName: "kube-api-access-lnw9p") pod "9423c4c1-2af6-46d6-86e6-7df836abe290" (UID: "9423c4c1-2af6-46d6-86e6-7df836abe290"). InnerVolumeSpecName "kube-api-access-lnw9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:43:47.921694 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:47.921655    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q47b\" (UniqueName: \"kubernetes.io/projected/face956b-dd2e-4bd3-8a43-a5c9f875f25e-kube-api-access-7q47b\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:43:47.921694 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:47.921686    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9423c4c1-2af6-46d6-86e6-7df836abe290-plugins-config-volume\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:43:47.921694 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:47.921700    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnw9p\" (UniqueName: \"kubernetes.io/projected/9423c4c1-2af6-46d6-86e6-7df836abe290-kube-api-access-lnw9p\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:43:47.921939 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:47.921714    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/face956b-dd2e-4bd3-8a43-a5c9f875f25e-plugins-config-volume\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:43:48.604672 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.604639    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm"
Sep 18 13:43:48.605137 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.604661    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm" event={"ID":"face956b-dd2e-4bd3-8a43-a5c9f875f25e","Type":"ContainerDied","Data":"8486eb9d57a5239de63d4e0315948dd25509dc4d9f5feac6b3b436bfde79216e"}
Sep 18 13:43:48.605137 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.604737    2566 scope.go:117] "RemoveContainer" containerID="247d4c1f6ccebe306744dfed1032ae093a90ae507435ab49c8ba80df32644843"
Sep 18 13:43:48.606226 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.606194    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h" event={"ID":"9423c4c1-2af6-46d6-86e6-7df836abe290","Type":"ContainerDied","Data":"fbad7ab45a0a23b484b140a4f79cdb566411375ed5efd7ea73008004165caf25"}
Sep 18 13:43:48.606226 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.606219    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h"
Sep 18 13:43:48.614541 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.614516    2566 scope.go:117] "RemoveContainer" containerID="40669338e30fa19dbf07994c92349bcced44fa1af0053476c78b580f30f7187d"
Sep 18 13:43:48.637704 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.637669    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm"]
Sep 18 13:43:48.650061 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.650032    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57732b-7b4c447bdf-9nvbm"]
Sep 18 13:43:48.675329 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.675294    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h"]
Sep 18 13:43:48.680691 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.680660    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57732b-5bf77956f8-kpk8h"]
Sep 18 13:43:48.880152 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.880072    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-951469-9q2sq"]
Sep 18 13:43:48.880511 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.880499    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="face956b-dd2e-4bd3-8a43-a5c9f875f25e" containerName="payload-processing"
Sep 18 13:43:48.880551 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.880513    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="face956b-dd2e-4bd3-8a43-a5c9f875f25e" containerName="payload-processing"
Sep 18 13:43:48.880551 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.880529    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9423c4c1-2af6-46d6-86e6-7df836abe290" containerName="payload-pre-processing"
Sep 18 13:43:48.880551 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.880535    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9423c4c1-2af6-46d6-86e6-7df836abe290" containerName="payload-pre-processing"
Sep 18 13:43:48.880551 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.880547    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40" containerName="payload-pre-processing"
Sep 18 13:43:48.880714 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.880554    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40" containerName="payload-pre-processing"
Sep 18 13:43:48.880714 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.880563    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c62fb1-9b10-434b-a9a9-bb15de4958e7" containerName="revoke-keys"
Sep 18 13:43:48.880714 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.880568    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c62fb1-9b10-434b-a9a9-bb15de4958e7" containerName="revoke-keys"
Sep 18 13:43:48.880714 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.880664    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b84a87a4-ca6b-4f24-ac8c-8b8bd2928a40" containerName="payload-pre-processing"
Sep 18 13:43:48.880714 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.880674    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9423c4c1-2af6-46d6-86e6-7df836abe290" containerName="payload-pre-processing"
Sep 18 13:43:48.880714 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.880681    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="face956b-dd2e-4bd3-8a43-a5c9f875f25e" containerName="payload-processing"
Sep 18 13:43:48.880714 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.880688    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="02c62fb1-9b10-434b-a9a9-bb15de4958e7" containerName="revoke-keys"
Sep 18 13:43:48.885365 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.885339    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-951469-9q2sq"
Sep 18 13:43:48.892679 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.892658    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:43:48.892817 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.892759    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nh2cd\""
Sep 18 13:43:48.923826 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.923793    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-951469-9q2sq"]
Sep 18 13:43:48.936706 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:48.936659    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa415372-2fcd-4394-9976-f73d871b2cad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-951469-9q2sq\" (UID: \"fa415372-2fcd-4394-9976-f73d871b2cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-951469-9q2sq"
Sep 18 13:43:49.037463 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:49.037425    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa415372-2fcd-4394-9976-f73d871b2cad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-951469-9q2sq\" (UID: \"fa415372-2fcd-4394-9976-f73d871b2cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-951469-9q2sq"
Sep 18 13:43:49.038087 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:49.038067    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa415372-2fcd-4394-9976-f73d871b2cad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-951469-9q2sq\" (UID: \"fa415372-2fcd-4394-9976-f73d871b2cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-951469-9q2sq"
Sep 18 13:43:49.196018 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:49.195918    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-951469-9q2sq"
Sep 18 13:43:49.486982 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:49.486956    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-951469-9q2sq"]
Sep 18 13:43:49.489636 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:43:49.489604    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa415372_2fcd_4394_9976_f73d871b2cad.slice/crio-a460309e46854319d934aeca8498fe85a1dfea8c2efc0d597bc0e59239a93e13 WatchSource:0}: Error finding container a460309e46854319d934aeca8498fe85a1dfea8c2efc0d597bc0e59239a93e13: Status 404 returned error can't find the container with id a460309e46854319d934aeca8498fe85a1dfea8c2efc0d597bc0e59239a93e13
Sep 18 13:43:49.614339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:49.614304    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-951469-9q2sq" event={"ID":"fa415372-2fcd-4394-9976-f73d871b2cad","Type":"ContainerStarted","Data":"a460309e46854319d934aeca8498fe85a1dfea8c2efc0d597bc0e59239a93e13"}
Sep 18 13:43:50.078174 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:50.078091    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9423c4c1-2af6-46d6-86e6-7df836abe290" path="/var/lib/kubelet/pods/9423c4c1-2af6-46d6-86e6-7df836abe290/volumes"
Sep 18 13:43:50.078506 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:50.078490    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="face956b-dd2e-4bd3-8a43-a5c9f875f25e" path="/var/lib/kubelet/pods/face956b-dd2e-4bd3-8a43-a5c9f875f25e/volumes"
Sep 18 13:43:50.622679 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:50.622645    2566 generic.go:356] "Generic (PLEG): container finished" podID="fa415372-2fcd-4394-9976-f73d871b2cad" containerID="48278f777fb9c76e47a9c9902d56ae775b5ff727a4cb6f5c24e58aa4f457415d" exitCode=0
Sep 18 13:43:50.623059 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:50.622712    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-951469-9q2sq" event={"ID":"fa415372-2fcd-4394-9976-f73d871b2cad","Type":"ContainerDied","Data":"48278f777fb9c76e47a9c9902d56ae775b5ff727a4cb6f5c24e58aa4f457415d"}
Sep 18 13:43:51.773564 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:51.773538    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-951469-9q2sq"
Sep 18 13:43:51.865899 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:51.865863    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa415372-2fcd-4394-9976-f73d871b2cad-maas-api-service-ca\") pod \"fa415372-2fcd-4394-9976-f73d871b2cad\" (UID: \"fa415372-2fcd-4394-9976-f73d871b2cad\") "
Sep 18 13:43:51.866240 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:51.866218    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa415372-2fcd-4394-9976-f73d871b2cad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fa415372-2fcd-4394-9976-f73d871b2cad" (UID: "fa415372-2fcd-4394-9976-f73d871b2cad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:43:51.967717 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:51.967640    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa415372-2fcd-4394-9976-f73d871b2cad-maas-api-service-ca\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:43:52.635324 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:52.635244    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-951469-9q2sq"
Sep 18 13:43:52.635324 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:52.635260    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-951469-9q2sq" event={"ID":"fa415372-2fcd-4394-9976-f73d871b2cad","Type":"ContainerDied","Data":"a460309e46854319d934aeca8498fe85a1dfea8c2efc0d597bc0e59239a93e13"}
Sep 18 13:43:52.635517 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:52.635339    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a460309e46854319d934aeca8498fe85a1dfea8c2efc0d597bc0e59239a93e13"
Sep 18 13:43:53.741211 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:53.741173    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"]
Sep 18 13:43:53.761542 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:53.761509    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"]
Sep 18 13:43:53.781266 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:43:53.781224    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"]
Sep 18 13:44:08.761060 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:08.761019    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq" podUID="9a8735a2-d52e-4781-bff2-6b516ecf5cec" containerName="main" containerID="cri-o://00f8e80fcccc457c2cf7f3784e5741e94157cbbd74cbc4e6b924882fa498159f" gracePeriod=45
Sep 18 13:44:08.781584 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:08.781548    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh" podUID="50fc70eb-512e-42d0-89ef-f91eb7e00e99" containerName="main" containerID="cri-o://260ba8d000110c641f78d83592e24d2f7cd829c85f9d6925887b577ea121b864" gracePeriod=45
Sep 18 13:44:08.801950 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:08.801881    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf" podUID="d6ad330a-f565-4baf-827e-535cb5864563" containerName="main" containerID="cri-o://e8297b776afd9ef04ef031727bee2ac2692d908bb2da8f598aef641b509c0c33" gracePeriod=45
Sep 18 13:44:09.076161 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.076135    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:44:09.105316 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.105292    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:44:09.108242 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.108223    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:44:09.254891 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.254864    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a8735a2-d52e-4781-bff2-6b516ecf5cec-tls-certs\") pod \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\" (UID: \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\") "
Sep 18 13:44:09.255051 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.254904    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50fc70eb-512e-42d0-89ef-f91eb7e00e99-dshm\") pod \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\" (UID: \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\") "
Sep 18 13:44:09.255051 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.254920    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50fc70eb-512e-42d0-89ef-f91eb7e00e99-model-cache\") pod \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\" (UID: \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\") "
Sep 18 13:44:09.255051 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.254947    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50fc70eb-512e-42d0-89ef-f91eb7e00e99-home\") pod \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\" (UID: \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\") "
Sep 18 13:44:09.255051 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.254977    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a8735a2-d52e-4781-bff2-6b516ecf5cec-home\") pod \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\" (UID: \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\") "
Sep 18 13:44:09.255051 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255004    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjwgv\" (UniqueName: \"kubernetes.io/projected/d6ad330a-f565-4baf-827e-535cb5864563-kube-api-access-fjwgv\") pod \"d6ad330a-f565-4baf-827e-535cb5864563\" (UID: \"d6ad330a-f565-4baf-827e-535cb5864563\") "
Sep 18 13:44:09.255051 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255022    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fc70eb-512e-42d0-89ef-f91eb7e00e99-tmp-dir\") pod \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\" (UID: \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\") "
Sep 18 13:44:09.255051 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255045    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ad330a-f565-4baf-827e-535cb5864563-tmp-dir\") pod \"d6ad330a-f565-4baf-827e-535cb5864563\" (UID: \"d6ad330a-f565-4baf-827e-535cb5864563\") "
Sep 18 13:44:09.255479 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255080    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50fc70eb-512e-42d0-89ef-f91eb7e00e99-tls-certs\") pod \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\" (UID: \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\") "
Sep 18 13:44:09.255479 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255125    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7phkd\" (UniqueName: \"kubernetes.io/projected/50fc70eb-512e-42d0-89ef-f91eb7e00e99-kube-api-access-7phkd\") pod \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\" (UID: \"50fc70eb-512e-42d0-89ef-f91eb7e00e99\") "
Sep 18 13:44:09.255479 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255155    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a8735a2-d52e-4781-bff2-6b516ecf5cec-tmp-dir\") pod \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\" (UID: \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\") "
Sep 18 13:44:09.255479 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255187    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fc70eb-512e-42d0-89ef-f91eb7e00e99-model-cache" (OuterVolumeSpecName: "model-cache") pod "50fc70eb-512e-42d0-89ef-f91eb7e00e99" (UID: "50fc70eb-512e-42d0-89ef-f91eb7e00e99"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:09.255479 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255202    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a8735a2-d52e-4781-bff2-6b516ecf5cec-model-cache\") pod \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\" (UID: \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\") "
Sep 18 13:44:09.255479 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255243    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8735a2-d52e-4781-bff2-6b516ecf5cec-home" (OuterVolumeSpecName: "home") pod "9a8735a2-d52e-4781-bff2-6b516ecf5cec" (UID: "9a8735a2-d52e-4781-bff2-6b516ecf5cec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:09.255479 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255245    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fc70eb-512e-42d0-89ef-f91eb7e00e99-home" (OuterVolumeSpecName: "home") pod "50fc70eb-512e-42d0-89ef-f91eb7e00e99" (UID: "50fc70eb-512e-42d0-89ef-f91eb7e00e99"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:09.255479 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255300    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ad330a-f565-4baf-827e-535cb5864563-dshm\") pod \"d6ad330a-f565-4baf-827e-535cb5864563\" (UID: \"d6ad330a-f565-4baf-827e-535cb5864563\") "
Sep 18 13:44:09.255479 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255338    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ad330a-f565-4baf-827e-535cb5864563-home\") pod \"d6ad330a-f565-4baf-827e-535cb5864563\" (UID: \"d6ad330a-f565-4baf-827e-535cb5864563\") "
Sep 18 13:44:09.255479 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255334    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ad330a-f565-4baf-827e-535cb5864563-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d6ad330a-f565-4baf-827e-535cb5864563" (UID: "d6ad330a-f565-4baf-827e-535cb5864563"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:09.255479 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255362    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a8735a2-d52e-4781-bff2-6b516ecf5cec-dshm\") pod \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\" (UID: \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\") "
Sep 18 13:44:09.255479 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255385    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8735a2-d52e-4781-bff2-6b516ecf5cec-model-cache" (OuterVolumeSpecName: "model-cache") pod "9a8735a2-d52e-4781-bff2-6b516ecf5cec" (UID: "9a8735a2-d52e-4781-bff2-6b516ecf5cec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:09.255479 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255394    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ad330a-f565-4baf-827e-535cb5864563-model-cache\") pod \"d6ad330a-f565-4baf-827e-535cb5864563\" (UID: \"d6ad330a-f565-4baf-827e-535cb5864563\") "
Sep 18 13:44:09.255479 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255434    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ad330a-f565-4baf-827e-535cb5864563-tls-certs\") pod \"d6ad330a-f565-4baf-827e-535cb5864563\" (UID: \"d6ad330a-f565-4baf-827e-535cb5864563\") "
Sep 18 13:44:09.255479 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255484    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5jsq\" (UniqueName: \"kubernetes.io/projected/9a8735a2-d52e-4781-bff2-6b516ecf5cec-kube-api-access-f5jsq\") pod \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\" (UID: \"9a8735a2-d52e-4781-bff2-6b516ecf5cec\") "
Sep 18 13:44:09.256303 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255561    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fc70eb-512e-42d0-89ef-f91eb7e00e99-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "50fc70eb-512e-42d0-89ef-f91eb7e00e99" (UID: "50fc70eb-512e-42d0-89ef-f91eb7e00e99"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:09.256303 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255575    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8735a2-d52e-4781-bff2-6b516ecf5cec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9a8735a2-d52e-4781-bff2-6b516ecf5cec" (UID: "9a8735a2-d52e-4781-bff2-6b516ecf5cec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:09.256303 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255804    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ad330a-f565-4baf-827e-535cb5864563-model-cache" (OuterVolumeSpecName: "model-cache") pod "d6ad330a-f565-4baf-827e-535cb5864563" (UID: "d6ad330a-f565-4baf-827e-535cb5864563"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:09.256303 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255940    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50fc70eb-512e-42d0-89ef-f91eb7e00e99-home\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:09.256303 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255957    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a8735a2-d52e-4781-bff2-6b516ecf5cec-home\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:09.256303 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255972    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fc70eb-512e-42d0-89ef-f91eb7e00e99-tmp-dir\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:09.256303 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.255988    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ad330a-f565-4baf-827e-535cb5864563-tmp-dir\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:09.256303 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.256000    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a8735a2-d52e-4781-bff2-6b516ecf5cec-tmp-dir\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:09.256303 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.256012    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a8735a2-d52e-4781-bff2-6b516ecf5cec-model-cache\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:09.256303 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.256026    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ad330a-f565-4baf-827e-535cb5864563-model-cache\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:09.256303 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.256037    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50fc70eb-512e-42d0-89ef-f91eb7e00e99-model-cache\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:09.257727 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.257697    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50fc70eb-512e-42d0-89ef-f91eb7e00e99-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "50fc70eb-512e-42d0-89ef-f91eb7e00e99" (UID: "50fc70eb-512e-42d0-89ef-f91eb7e00e99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:44:09.257851 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.257780    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8735a2-d52e-4781-bff2-6b516ecf5cec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9a8735a2-d52e-4781-bff2-6b516ecf5cec" (UID: "9a8735a2-d52e-4781-bff2-6b516ecf5cec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:44:09.257851 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.257811    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ad330a-f565-4baf-827e-535cb5864563-kube-api-access-fjwgv" (OuterVolumeSpecName: "kube-api-access-fjwgv") pod "d6ad330a-f565-4baf-827e-535cb5864563" (UID: "d6ad330a-f565-4baf-827e-535cb5864563"). InnerVolumeSpecName "kube-api-access-fjwgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:44:09.258048 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.258016    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ad330a-f565-4baf-827e-535cb5864563-home" (OuterVolumeSpecName: "home") pod "d6ad330a-f565-4baf-827e-535cb5864563" (UID: "d6ad330a-f565-4baf-827e-535cb5864563"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:09.258149 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.258047    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8735a2-d52e-4781-bff2-6b516ecf5cec-kube-api-access-f5jsq" (OuterVolumeSpecName: "kube-api-access-f5jsq") pod "9a8735a2-d52e-4781-bff2-6b516ecf5cec" (UID: "9a8735a2-d52e-4781-bff2-6b516ecf5cec"). InnerVolumeSpecName "kube-api-access-f5jsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:44:09.258280 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.258257    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50fc70eb-512e-42d0-89ef-f91eb7e00e99-kube-api-access-7phkd" (OuterVolumeSpecName: "kube-api-access-7phkd") pod "50fc70eb-512e-42d0-89ef-f91eb7e00e99" (UID: "50fc70eb-512e-42d0-89ef-f91eb7e00e99"). InnerVolumeSpecName "kube-api-access-7phkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:44:09.258322 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.258276    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8735a2-d52e-4781-bff2-6b516ecf5cec-dshm" (OuterVolumeSpecName: "dshm") pod "9a8735a2-d52e-4781-bff2-6b516ecf5cec" (UID: "9a8735a2-d52e-4781-bff2-6b516ecf5cec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:09.258395 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.258381    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6ad330a-f565-4baf-827e-535cb5864563-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6ad330a-f565-4baf-827e-535cb5864563" (UID: "d6ad330a-f565-4baf-827e-535cb5864563"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:44:09.258995 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.258977    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fc70eb-512e-42d0-89ef-f91eb7e00e99-dshm" (OuterVolumeSpecName: "dshm") pod "50fc70eb-512e-42d0-89ef-f91eb7e00e99" (UID: "50fc70eb-512e-42d0-89ef-f91eb7e00e99"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:09.259484 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.259465    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ad330a-f565-4baf-827e-535cb5864563-dshm" (OuterVolumeSpecName: "dshm") pod "d6ad330a-f565-4baf-827e-535cb5864563" (UID: "d6ad330a-f565-4baf-827e-535cb5864563"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:09.357128 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.357094    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjwgv\" (UniqueName: \"kubernetes.io/projected/d6ad330a-f565-4baf-827e-535cb5864563-kube-api-access-fjwgv\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:09.357128 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.357126    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50fc70eb-512e-42d0-89ef-f91eb7e00e99-tls-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:09.357326 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.357137    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7phkd\" (UniqueName: \"kubernetes.io/projected/50fc70eb-512e-42d0-89ef-f91eb7e00e99-kube-api-access-7phkd\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:09.357326 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.357149    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ad330a-f565-4baf-827e-535cb5864563-dshm\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:09.357326 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.357159    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ad330a-f565-4baf-827e-535cb5864563-home\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:09.357326 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.357167    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a8735a2-d52e-4781-bff2-6b516ecf5cec-dshm\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:09.357326 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.357174    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ad330a-f565-4baf-827e-535cb5864563-tls-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:09.357326 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.357183    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5jsq\" (UniqueName: \"kubernetes.io/projected/9a8735a2-d52e-4781-bff2-6b516ecf5cec-kube-api-access-f5jsq\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:09.357326 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.357191    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a8735a2-d52e-4781-bff2-6b516ecf5cec-tls-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:09.357326 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.357199    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50fc70eb-512e-42d0-89ef-f91eb7e00e99-dshm\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:09.731864 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.731772    2566 generic.go:356] "Generic (PLEG): container finished" podID="50fc70eb-512e-42d0-89ef-f91eb7e00e99" containerID="260ba8d000110c641f78d83592e24d2f7cd829c85f9d6925887b577ea121b864" exitCode=0
Sep 18 13:44:09.731864 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.731824    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"
Sep 18 13:44:09.731864 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.731850    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh" event={"ID":"50fc70eb-512e-42d0-89ef-f91eb7e00e99","Type":"ContainerDied","Data":"260ba8d000110c641f78d83592e24d2f7cd829c85f9d6925887b577ea121b864"}
Sep 18 13:44:09.732123 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.731895    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh" event={"ID":"50fc70eb-512e-42d0-89ef-f91eb7e00e99","Type":"ContainerDied","Data":"a6fea2103acd53c11b63bb22487e9fc623f638754a318d81935fadadf4c4c0ce"}
Sep 18 13:44:09.732123 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.731918    2566 scope.go:117] "RemoveContainer" containerID="260ba8d000110c641f78d83592e24d2f7cd829c85f9d6925887b577ea121b864"
Sep 18 13:44:09.733285 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.733262    2566 generic.go:356] "Generic (PLEG): container finished" podID="9a8735a2-d52e-4781-bff2-6b516ecf5cec" containerID="00f8e80fcccc457c2cf7f3784e5741e94157cbbd74cbc4e6b924882fa498159f" exitCode=0
Sep 18 13:44:09.733416 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.733325    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"
Sep 18 13:44:09.733416 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.733341    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq" event={"ID":"9a8735a2-d52e-4781-bff2-6b516ecf5cec","Type":"ContainerDied","Data":"00f8e80fcccc457c2cf7f3784e5741e94157cbbd74cbc4e6b924882fa498159f"}
Sep 18 13:44:09.733416 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.733378    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq" event={"ID":"9a8735a2-d52e-4781-bff2-6b516ecf5cec","Type":"ContainerDied","Data":"8f2aa240584b16c8d4d6d79a2b3a7422a9d199b4a16a002b5b7756f1f4927139"}
Sep 18 13:44:09.734776 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.734754    2566 generic.go:356] "Generic (PLEG): container finished" podID="d6ad330a-f565-4baf-827e-535cb5864563" containerID="e8297b776afd9ef04ef031727bee2ac2692d908bb2da8f598aef641b509c0c33" exitCode=0
Sep 18 13:44:09.734880 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.734797    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"
Sep 18 13:44:09.734880 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.734833    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf" event={"ID":"d6ad330a-f565-4baf-827e-535cb5864563","Type":"ContainerDied","Data":"e8297b776afd9ef04ef031727bee2ac2692d908bb2da8f598aef641b509c0c33"}
Sep 18 13:44:09.734880 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.734862    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf" event={"ID":"d6ad330a-f565-4baf-827e-535cb5864563","Type":"ContainerDied","Data":"13befb80734a041456fdecdefb945b7d5cba96445c5fe7ccff96728a80186869"}
Sep 18 13:44:09.742205 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.742178    2566 scope.go:117] "RemoveContainer" containerID="260ba8d000110c641f78d83592e24d2f7cd829c85f9d6925887b577ea121b864"
Sep 18 13:44:09.742566 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:44:09.742532    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"260ba8d000110c641f78d83592e24d2f7cd829c85f9d6925887b577ea121b864\": container with ID starting with 260ba8d000110c641f78d83592e24d2f7cd829c85f9d6925887b577ea121b864 not found: ID does not exist" containerID="260ba8d000110c641f78d83592e24d2f7cd829c85f9d6925887b577ea121b864"
Sep 18 13:44:09.742697 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.742571    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"260ba8d000110c641f78d83592e24d2f7cd829c85f9d6925887b577ea121b864"} err="failed to get container status \"260ba8d000110c641f78d83592e24d2f7cd829c85f9d6925887b577ea121b864\": rpc error: code = NotFound desc = could not find container \"260ba8d000110c641f78d83592e24d2f7cd829c85f9d6925887b577ea121b864\": container with ID starting with 260ba8d000110c641f78d83592e24d2f7cd829c85f9d6925887b577ea121b864 not found: ID does not exist"
Sep 18 13:44:09.742697 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.742612    2566 scope.go:117] "RemoveContainer" containerID="00f8e80fcccc457c2cf7f3784e5741e94157cbbd74cbc4e6b924882fa498159f"
Sep 18 13:44:09.751851 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.751835    2566 scope.go:117] "RemoveContainer" containerID="00f8e80fcccc457c2cf7f3784e5741e94157cbbd74cbc4e6b924882fa498159f"
Sep 18 13:44:09.752107 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:44:09.752089    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00f8e80fcccc457c2cf7f3784e5741e94157cbbd74cbc4e6b924882fa498159f\": container with ID starting with 00f8e80fcccc457c2cf7f3784e5741e94157cbbd74cbc4e6b924882fa498159f not found: ID does not exist" containerID="00f8e80fcccc457c2cf7f3784e5741e94157cbbd74cbc4e6b924882fa498159f"
Sep 18 13:44:09.752160 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.752115    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00f8e80fcccc457c2cf7f3784e5741e94157cbbd74cbc4e6b924882fa498159f"} err="failed to get container status \"00f8e80fcccc457c2cf7f3784e5741e94157cbbd74cbc4e6b924882fa498159f\": rpc error: code = NotFound desc = could not find container \"00f8e80fcccc457c2cf7f3784e5741e94157cbbd74cbc4e6b924882fa498159f\": container with ID starting with 00f8e80fcccc457c2cf7f3784e5741e94157cbbd74cbc4e6b924882fa498159f not found: ID does not exist"
Sep 18 13:44:09.752160 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.752131    2566 scope.go:117] "RemoveContainer" containerID="e8297b776afd9ef04ef031727bee2ac2692d908bb2da8f598aef641b509c0c33"
Sep 18 13:44:09.759926 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.759907    2566 scope.go:117] "RemoveContainer" containerID="e8297b776afd9ef04ef031727bee2ac2692d908bb2da8f598aef641b509c0c33"
Sep 18 13:44:09.760202 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:44:09.760182    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8297b776afd9ef04ef031727bee2ac2692d908bb2da8f598aef641b509c0c33\": container with ID starting with e8297b776afd9ef04ef031727bee2ac2692d908bb2da8f598aef641b509c0c33 not found: ID does not exist" containerID="e8297b776afd9ef04ef031727bee2ac2692d908bb2da8f598aef641b509c0c33"
Sep 18 13:44:09.760254 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.760210    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8297b776afd9ef04ef031727bee2ac2692d908bb2da8f598aef641b509c0c33"} err="failed to get container status \"e8297b776afd9ef04ef031727bee2ac2692d908bb2da8f598aef641b509c0c33\": rpc error: code = NotFound desc = could not find container \"e8297b776afd9ef04ef031727bee2ac2692d908bb2da8f598aef641b509c0c33\": container with ID starting with e8297b776afd9ef04ef031727bee2ac2692d908bb2da8f598aef641b509c0c33 not found: ID does not exist"
Sep 18 13:44:09.779530 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.779505    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"]
Sep 18 13:44:09.781876 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.781849    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-951469/distinct-2-w0-951469-kserve-77d78557b6-pjmbq"]
Sep 18 13:44:09.806511 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.806481    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"]
Sep 18 13:44:09.819031 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.819005    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-951469/embedding-w0-951469-kserve-7b8f6b4c5d-rsrhf"]
Sep 18 13:44:09.835817 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.835791    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"]
Sep 18 13:44:09.842761 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:09.842738    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-951469/distinct-w0-951469-kserve-dbbc8fb6f-q49lh"]
Sep 18 13:44:10.078429 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:10.078348    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50fc70eb-512e-42d0-89ef-f91eb7e00e99" path="/var/lib/kubelet/pods/50fc70eb-512e-42d0-89ef-f91eb7e00e99/volumes"
Sep 18 13:44:10.078804 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:10.078789    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a8735a2-d52e-4781-bff2-6b516ecf5cec" path="/var/lib/kubelet/pods/9a8735a2-d52e-4781-bff2-6b516ecf5cec/volumes"
Sep 18 13:44:10.079164 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:10.079152    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6ad330a-f565-4baf-827e-535cb5864563" path="/var/lib/kubelet/pods/d6ad330a-f565-4baf-827e-535cb5864563/volumes"
Sep 18 13:44:10.556132 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:10.556099    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-951469-9q2sq"]
Sep 18 13:44:10.589836 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:10.589772    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-951469-9q2sq"]
Sep 18 13:44:12.077074 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:12.077041    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa415372-2fcd-4394-9976-f73d871b2cad" path="/var/lib/kubelet/pods/fa415372-2fcd-4394-9976-f73d871b2cad/volumes"
Sep 18 13:44:19.451660 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:19.451617    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"]
Sep 18 13:44:19.452083 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:19.451842    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9" podUID="26a1a326-b1b7-4266-9907-d729cabc4d58" containerName="istio-proxy" containerID="cri-o://dc6ac75ebbd7b1f5f079deb52058e87498e767d6f3a2a5dfd9387e760d304475" gracePeriod=30
Sep 18 13:44:20.202305 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:20.202269    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"]
Sep 18 13:44:20.202497 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:20.202473    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg" podUID="17414e82-b0f6-4822-9eda-0b7d0a567925" containerName="istio-proxy" containerID="cri-o://3c6f17d69627c95bf7e0c09da7be5983375d400a8e4cf0c06f9b3a41ab6e46de" gracePeriod=30
Sep 18 13:44:24.706083 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.706057    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:44:24.808004 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.807911    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-workload-certs\") pod \"26a1a326-b1b7-4266-9907-d729cabc4d58\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") "
Sep 18 13:44:24.808004 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.807976    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgpxx\" (UniqueName: \"kubernetes.io/projected/26a1a326-b1b7-4266-9907-d729cabc4d58-kube-api-access-pgpxx\") pod \"26a1a326-b1b7-4266-9907-d729cabc4d58\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") "
Sep 18 13:44:24.808238 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.808013    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-credential-socket\") pod \"26a1a326-b1b7-4266-9907-d729cabc4d58\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") "
Sep 18 13:44:24.808238 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.808037    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-istio-envoy\") pod \"26a1a326-b1b7-4266-9907-d729cabc4d58\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") "
Sep 18 13:44:24.808238 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.808090    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a1a326-b1b7-4266-9907-d729cabc4d58-istio-podinfo\") pod \"26a1a326-b1b7-4266-9907-d729cabc4d58\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") "
Sep 18 13:44:24.808238 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.808130    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-workload-socket\") pod \"26a1a326-b1b7-4266-9907-d729cabc4d58\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") "
Sep 18 13:44:24.808238 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.808186    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-istio-data\") pod \"26a1a326-b1b7-4266-9907-d729cabc4d58\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") "
Sep 18 13:44:24.808490 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.808215    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "26a1a326-b1b7-4266-9907-d729cabc4d58" (UID: "26a1a326-b1b7-4266-9907-d729cabc4d58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:24.808490 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.808262    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a1a326-b1b7-4266-9907-d729cabc4d58-istio-token\") pod \"26a1a326-b1b7-4266-9907-d729cabc4d58\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") "
Sep 18 13:44:24.808490 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.808273    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "26a1a326-b1b7-4266-9907-d729cabc4d58" (UID: "26a1a326-b1b7-4266-9907-d729cabc4d58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:24.808490 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.808291    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a1a326-b1b7-4266-9907-d729cabc4d58-istiod-ca-cert\") pod \"26a1a326-b1b7-4266-9907-d729cabc4d58\" (UID: \"26a1a326-b1b7-4266-9907-d729cabc4d58\") "
Sep 18 13:44:24.808490 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.808421    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-istio-data" (OuterVolumeSpecName: "istio-data") pod "26a1a326-b1b7-4266-9907-d729cabc4d58" (UID: "26a1a326-b1b7-4266-9907-d729cabc4d58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:24.808490 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.808445    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "26a1a326-b1b7-4266-9907-d729cabc4d58" (UID: "26a1a326-b1b7-4266-9907-d729cabc4d58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:24.808839 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.808678    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-workload-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:24.808839 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.808695    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a1a326-b1b7-4266-9907-d729cabc4d58-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "26a1a326-b1b7-4266-9907-d729cabc4d58" (UID: "26a1a326-b1b7-4266-9907-d729cabc4d58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:44:24.808839 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.808703    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-istio-data\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:24.808839 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.808743    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-workload-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:24.808839 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.808759    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-credential-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:24.810429 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.810396    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "26a1a326-b1b7-4266-9907-d729cabc4d58" (UID: "26a1a326-b1b7-4266-9907-d729cabc4d58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:24.810429 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.810407    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a1a326-b1b7-4266-9907-d729cabc4d58-kube-api-access-pgpxx" (OuterVolumeSpecName: "kube-api-access-pgpxx") pod "26a1a326-b1b7-4266-9907-d729cabc4d58" (UID: "26a1a326-b1b7-4266-9907-d729cabc4d58"). InnerVolumeSpecName "kube-api-access-pgpxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:44:24.810429 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.810421    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a1a326-b1b7-4266-9907-d729cabc4d58-istio-token" (OuterVolumeSpecName: "istio-token") pod "26a1a326-b1b7-4266-9907-d729cabc4d58" (UID: "26a1a326-b1b7-4266-9907-d729cabc4d58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:44:24.810429 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.810404    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26a1a326-b1b7-4266-9907-d729cabc4d58-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "26a1a326-b1b7-4266-9907-d729cabc4d58" (UID: "26a1a326-b1b7-4266-9907-d729cabc4d58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 13:44:24.824478 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.824449    2566 generic.go:356] "Generic (PLEG): container finished" podID="26a1a326-b1b7-4266-9907-d729cabc4d58" containerID="dc6ac75ebbd7b1f5f079deb52058e87498e767d6f3a2a5dfd9387e760d304475" exitCode=0
Sep 18 13:44:24.824615 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.824499    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9" event={"ID":"26a1a326-b1b7-4266-9907-d729cabc4d58","Type":"ContainerDied","Data":"dc6ac75ebbd7b1f5f079deb52058e87498e767d6f3a2a5dfd9387e760d304475"}
Sep 18 13:44:24.824615 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.824511    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"
Sep 18 13:44:24.824615 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.824544    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9" event={"ID":"26a1a326-b1b7-4266-9907-d729cabc4d58","Type":"ContainerDied","Data":"305b939715df7b25bf0b9933032e627cde016c1fee0f8c1c05c09062aef52719"}
Sep 18 13:44:24.824615 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.824570    2566 scope.go:117] "RemoveContainer" containerID="dc6ac75ebbd7b1f5f079deb52058e87498e767d6f3a2a5dfd9387e760d304475"
Sep 18 13:44:24.834312 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.834294    2566 scope.go:117] "RemoveContainer" containerID="9def3daef3c488564ab79620d8a2c52e4d0874e7f3a389b59e0c2cc44a66beb0"
Sep 18 13:44:24.842459 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.842443    2566 scope.go:117] "RemoveContainer" containerID="dc6ac75ebbd7b1f5f079deb52058e87498e767d6f3a2a5dfd9387e760d304475"
Sep 18 13:44:24.842781 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:44:24.842748    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc6ac75ebbd7b1f5f079deb52058e87498e767d6f3a2a5dfd9387e760d304475\": container with ID starting with dc6ac75ebbd7b1f5f079deb52058e87498e767d6f3a2a5dfd9387e760d304475 not found: ID does not exist" containerID="dc6ac75ebbd7b1f5f079deb52058e87498e767d6f3a2a5dfd9387e760d304475"
Sep 18 13:44:24.842849 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.842789    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc6ac75ebbd7b1f5f079deb52058e87498e767d6f3a2a5dfd9387e760d304475"} err="failed to get container status \"dc6ac75ebbd7b1f5f079deb52058e87498e767d6f3a2a5dfd9387e760d304475\": rpc error: code = NotFound desc = could not find container \"dc6ac75ebbd7b1f5f079deb52058e87498e767d6f3a2a5dfd9387e760d304475\": container with ID starting with dc6ac75ebbd7b1f5f079deb52058e87498e767d6f3a2a5dfd9387e760d304475 not found: ID does not exist"
Sep 18 13:44:24.842849 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.842805    2566 scope.go:117] "RemoveContainer" containerID="9def3daef3c488564ab79620d8a2c52e4d0874e7f3a389b59e0c2cc44a66beb0"
Sep 18 13:44:24.843059 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:44:24.843040    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9def3daef3c488564ab79620d8a2c52e4d0874e7f3a389b59e0c2cc44a66beb0\": container with ID starting with 9def3daef3c488564ab79620d8a2c52e4d0874e7f3a389b59e0c2cc44a66beb0 not found: ID does not exist" containerID="9def3daef3c488564ab79620d8a2c52e4d0874e7f3a389b59e0c2cc44a66beb0"
Sep 18 13:44:24.843111 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.843065    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9def3daef3c488564ab79620d8a2c52e4d0874e7f3a389b59e0c2cc44a66beb0"} err="failed to get container status \"9def3daef3c488564ab79620d8a2c52e4d0874e7f3a389b59e0c2cc44a66beb0\": rpc error: code = NotFound desc = could not find container \"9def3daef3c488564ab79620d8a2c52e4d0874e7f3a389b59e0c2cc44a66beb0\": container with ID starting with 9def3daef3c488564ab79620d8a2c52e4d0874e7f3a389b59e0c2cc44a66beb0 not found: ID does not exist"
Sep 18 13:44:24.859203 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.859179    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"]
Sep 18 13:44:24.864270 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.864246    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-951469-openshift-default-5dd5757f76-crzh9"]
Sep 18 13:44:24.909745 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.909711    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a1a326-b1b7-4266-9907-d729cabc4d58-istio-token\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:24.909745 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.909740    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a1a326-b1b7-4266-9907-d729cabc4d58-istiod-ca-cert\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:24.909745 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.909749    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgpxx\" (UniqueName: \"kubernetes.io/projected/26a1a326-b1b7-4266-9907-d729cabc4d58-kube-api-access-pgpxx\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:24.909964 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.909758    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a1a326-b1b7-4266-9907-d729cabc4d58-istio-envoy\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:24.909964 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:24.909768    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a1a326-b1b7-4266-9907-d729cabc4d58-istio-podinfo\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:25.443482 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.443456    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:44:25.515370 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.515339    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17414e82-b0f6-4822-9eda-0b7d0a567925-istiod-ca-cert\") pod \"17414e82-b0f6-4822-9eda-0b7d0a567925\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") "
Sep 18 13:44:25.515521 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.515424    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17414e82-b0f6-4822-9eda-0b7d0a567925-istio-token\") pod \"17414e82-b0f6-4822-9eda-0b7d0a567925\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") "
Sep 18 13:44:25.515521 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.515457    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-workload-socket\") pod \"17414e82-b0f6-4822-9eda-0b7d0a567925\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") "
Sep 18 13:44:25.515521 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.515478    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-istio-data\") pod \"17414e82-b0f6-4822-9eda-0b7d0a567925\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") "
Sep 18 13:44:25.515521 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.515497    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17414e82-b0f6-4822-9eda-0b7d0a567925-istio-podinfo\") pod \"17414e82-b0f6-4822-9eda-0b7d0a567925\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") "
Sep 18 13:44:25.515779 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.515578    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gf58\" (UniqueName: \"kubernetes.io/projected/17414e82-b0f6-4822-9eda-0b7d0a567925-kube-api-access-6gf58\") pod \"17414e82-b0f6-4822-9eda-0b7d0a567925\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") "
Sep 18 13:44:25.515779 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.515676    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-credential-socket\") pod \"17414e82-b0f6-4822-9eda-0b7d0a567925\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") "
Sep 18 13:44:25.515779 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.515740    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-istio-envoy\") pod \"17414e82-b0f6-4822-9eda-0b7d0a567925\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") "
Sep 18 13:44:25.515779 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.515748    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "17414e82-b0f6-4822-9eda-0b7d0a567925" (UID: "17414e82-b0f6-4822-9eda-0b7d0a567925"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:25.515985 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.515773    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-workload-certs\") pod \"17414e82-b0f6-4822-9eda-0b7d0a567925\" (UID: \"17414e82-b0f6-4822-9eda-0b7d0a567925\") "
Sep 18 13:44:25.515985 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.515829    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-istio-data" (OuterVolumeSpecName: "istio-data") pod "17414e82-b0f6-4822-9eda-0b7d0a567925" (UID: "17414e82-b0f6-4822-9eda-0b7d0a567925"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:25.515985 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.515847    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17414e82-b0f6-4822-9eda-0b7d0a567925-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "17414e82-b0f6-4822-9eda-0b7d0a567925" (UID: "17414e82-b0f6-4822-9eda-0b7d0a567925"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:44:25.516135 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.515987    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "17414e82-b0f6-4822-9eda-0b7d0a567925" (UID: "17414e82-b0f6-4822-9eda-0b7d0a567925"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:25.516135 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.516073    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "17414e82-b0f6-4822-9eda-0b7d0a567925" (UID: "17414e82-b0f6-4822-9eda-0b7d0a567925"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:25.516257 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.516239    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-workload-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:25.516321 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.516262    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-istio-data\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:25.516321 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.516279    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-credential-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:25.516321 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.516293    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-workload-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:25.516321 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.516306    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17414e82-b0f6-4822-9eda-0b7d0a567925-istiod-ca-cert\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:25.518371 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.518281    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17414e82-b0f6-4822-9eda-0b7d0a567925-istio-token" (OuterVolumeSpecName: "istio-token") pod "17414e82-b0f6-4822-9eda-0b7d0a567925" (UID: "17414e82-b0f6-4822-9eda-0b7d0a567925"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:44:25.518371 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.518301    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17414e82-b0f6-4822-9eda-0b7d0a567925-kube-api-access-6gf58" (OuterVolumeSpecName: "kube-api-access-6gf58") pod "17414e82-b0f6-4822-9eda-0b7d0a567925" (UID: "17414e82-b0f6-4822-9eda-0b7d0a567925"). InnerVolumeSpecName "kube-api-access-6gf58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:44:25.518578 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.518373    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "17414e82-b0f6-4822-9eda-0b7d0a567925" (UID: "17414e82-b0f6-4822-9eda-0b7d0a567925"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:25.527397 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.527356    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17414e82-b0f6-4822-9eda-0b7d0a567925-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "17414e82-b0f6-4822-9eda-0b7d0a567925" (UID: "17414e82-b0f6-4822-9eda-0b7d0a567925"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 13:44:25.617252 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.617216    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17414e82-b0f6-4822-9eda-0b7d0a567925-istio-token\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:25.617252 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.617246    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17414e82-b0f6-4822-9eda-0b7d0a567925-istio-podinfo\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:25.617252 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.617256    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gf58\" (UniqueName: \"kubernetes.io/projected/17414e82-b0f6-4822-9eda-0b7d0a567925-kube-api-access-6gf58\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:25.617494 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.617267    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17414e82-b0f6-4822-9eda-0b7d0a567925-istio-envoy\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:25.831628 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.831525    2566 generic.go:356] "Generic (PLEG): container finished" podID="17414e82-b0f6-4822-9eda-0b7d0a567925" containerID="3c6f17d69627c95bf7e0c09da7be5983375d400a8e4cf0c06f9b3a41ab6e46de" exitCode=0
Sep 18 13:44:25.831628 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.831585    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"
Sep 18 13:44:25.831628 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.831611    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg" event={"ID":"17414e82-b0f6-4822-9eda-0b7d0a567925","Type":"ContainerDied","Data":"3c6f17d69627c95bf7e0c09da7be5983375d400a8e4cf0c06f9b3a41ab6e46de"}
Sep 18 13:44:25.832163 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.831695    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg" event={"ID":"17414e82-b0f6-4822-9eda-0b7d0a567925","Type":"ContainerDied","Data":"61abd110a3688093638c3870b5494b276970effcf010fcd1ccb8fcd723e69fe3"}
Sep 18 13:44:25.832163 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.831719    2566 scope.go:117] "RemoveContainer" containerID="3c6f17d69627c95bf7e0c09da7be5983375d400a8e4cf0c06f9b3a41ab6e46de"
Sep 18 13:44:25.842387 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.842367    2566 scope.go:117] "RemoveContainer" containerID="2f279ca03a1c427ffb0d5114a54207e46ae3125c9bbc61a01b35df191e67052c"
Sep 18 13:44:25.850601 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.850572    2566 scope.go:117] "RemoveContainer" containerID="3c6f17d69627c95bf7e0c09da7be5983375d400a8e4cf0c06f9b3a41ab6e46de"
Sep 18 13:44:25.850890 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:44:25.850867    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c6f17d69627c95bf7e0c09da7be5983375d400a8e4cf0c06f9b3a41ab6e46de\": container with ID starting with 3c6f17d69627c95bf7e0c09da7be5983375d400a8e4cf0c06f9b3a41ab6e46de not found: ID does not exist" containerID="3c6f17d69627c95bf7e0c09da7be5983375d400a8e4cf0c06f9b3a41ab6e46de"
Sep 18 13:44:25.850967 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.850894    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c6f17d69627c95bf7e0c09da7be5983375d400a8e4cf0c06f9b3a41ab6e46de"} err="failed to get container status \"3c6f17d69627c95bf7e0c09da7be5983375d400a8e4cf0c06f9b3a41ab6e46de\": rpc error: code = NotFound desc = could not find container \"3c6f17d69627c95bf7e0c09da7be5983375d400a8e4cf0c06f9b3a41ab6e46de\": container with ID starting with 3c6f17d69627c95bf7e0c09da7be5983375d400a8e4cf0c06f9b3a41ab6e46de not found: ID does not exist"
Sep 18 13:44:25.850967 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.850911    2566 scope.go:117] "RemoveContainer" containerID="2f279ca03a1c427ffb0d5114a54207e46ae3125c9bbc61a01b35df191e67052c"
Sep 18 13:44:25.851168 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:44:25.851149    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f279ca03a1c427ffb0d5114a54207e46ae3125c9bbc61a01b35df191e67052c\": container with ID starting with 2f279ca03a1c427ffb0d5114a54207e46ae3125c9bbc61a01b35df191e67052c not found: ID does not exist" containerID="2f279ca03a1c427ffb0d5114a54207e46ae3125c9bbc61a01b35df191e67052c"
Sep 18 13:44:25.851210 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.851174    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f279ca03a1c427ffb0d5114a54207e46ae3125c9bbc61a01b35df191e67052c"} err="failed to get container status \"2f279ca03a1c427ffb0d5114a54207e46ae3125c9bbc61a01b35df191e67052c\": rpc error: code = NotFound desc = could not find container \"2f279ca03a1c427ffb0d5114a54207e46ae3125c9bbc61a01b35df191e67052c\": container with ID starting with 2f279ca03a1c427ffb0d5114a54207e46ae3125c9bbc61a01b35df191e67052c not found: ID does not exist"
Sep 18 13:44:25.871392 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.871362    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"]
Sep 18 13:44:25.880309 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:25.880283    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b2d5da-openshift-default-6574867646-qxssg"]
Sep 18 13:44:26.077116 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:26.077082    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17414e82-b0f6-4822-9eda-0b7d0a567925" path="/var/lib/kubelet/pods/17414e82-b0f6-4822-9eda-0b7d0a567925/volumes"
Sep 18 13:44:26.077543 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:26.077530    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26a1a326-b1b7-4266-9907-d729cabc4d58" path="/var/lib/kubelet/pods/26a1a326-b1b7-4266-9907-d729cabc4d58/volumes"
Sep 18 13:44:42.047323 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:42.047286    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"]
Sep 18 13:44:42.047836 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:42.047525    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l" podUID="a9244fde-4e58-4ee0-a4ac-17827a4ac2dd" containerName="istio-proxy" containerID="cri-o://fc4f4b7e012d831333391208e1f2fc6c2e464b788888cc2e0a44a7cd85ab88ba" gracePeriod=30
Sep 18 13:44:47.305023 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.304997    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:44:47.429314 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.429277    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-credential-socket\") pod \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") "
Sep 18 13:44:47.429314 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.429312    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-workload-socket\") pod \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") "
Sep 18 13:44:47.429589 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.429380    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istio-envoy\") pod \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") "
Sep 18 13:44:47.429589 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.429407    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istiod-ca-cert\") pod \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") "
Sep 18 13:44:47.429589 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.429449    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istio-data\") pod \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") "
Sep 18 13:44:47.429589 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.429465    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istio-token\") pod \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") "
Sep 18 13:44:47.429589 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.429491    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnmkc\" (UniqueName: \"kubernetes.io/projected/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-kube-api-access-hnmkc\") pod \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") "
Sep 18 13:44:47.429589 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.429545    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-workload-certs\") pod \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") "
Sep 18 13:44:47.429971 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.429624    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a9244fde-4e58-4ee0-a4ac-17827a4ac2dd" (UID: "a9244fde-4e58-4ee0-a4ac-17827a4ac2dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:47.429971 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.429687    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istio-podinfo\") pod \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\" (UID: \"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd\") "
Sep 18 13:44:47.429971 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.429825    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a9244fde-4e58-4ee0-a4ac-17827a4ac2dd" (UID: "a9244fde-4e58-4ee0-a4ac-17827a4ac2dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:47.429971 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.429894    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a9244fde-4e58-4ee0-a4ac-17827a4ac2dd" (UID: "a9244fde-4e58-4ee0-a4ac-17827a4ac2dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:47.429971 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.429923    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a9244fde-4e58-4ee0-a4ac-17827a4ac2dd" (UID: "a9244fde-4e58-4ee0-a4ac-17827a4ac2dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:44:47.430205 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.430005    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istio-data" (OuterVolumeSpecName: "istio-data") pod "a9244fde-4e58-4ee0-a4ac-17827a4ac2dd" (UID: "a9244fde-4e58-4ee0-a4ac-17827a4ac2dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:47.430205 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.430134    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istiod-ca-cert\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:47.430205 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.430148    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istio-data\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:47.430205 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.430156    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-workload-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:47.430205 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.430164    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-credential-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:47.430205 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.430174    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-workload-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:47.431763 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.431737    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a9244fde-4e58-4ee0-a4ac-17827a4ac2dd" (UID: "a9244fde-4e58-4ee0-a4ac-17827a4ac2dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:47.431913 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.431880    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a9244fde-4e58-4ee0-a4ac-17827a4ac2dd" (UID: "a9244fde-4e58-4ee0-a4ac-17827a4ac2dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 13:44:47.432175 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.432154    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-kube-api-access-hnmkc" (OuterVolumeSpecName: "kube-api-access-hnmkc") pod "a9244fde-4e58-4ee0-a4ac-17827a4ac2dd" (UID: "a9244fde-4e58-4ee0-a4ac-17827a4ac2dd"). InnerVolumeSpecName "kube-api-access-hnmkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:44:47.432175 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.432163    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istio-token" (OuterVolumeSpecName: "istio-token") pod "a9244fde-4e58-4ee0-a4ac-17827a4ac2dd" (UID: "a9244fde-4e58-4ee0-a4ac-17827a4ac2dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:44:47.531114 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.531060    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istio-envoy\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:47.531114 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.531093    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istio-token\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:47.531114 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.531102    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnmkc\" (UniqueName: \"kubernetes.io/projected/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-kube-api-access-hnmkc\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:47.531114 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.531111    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd-istio-podinfo\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:44:47.954926 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.954893    2566 generic.go:356] "Generic (PLEG): container finished" podID="a9244fde-4e58-4ee0-a4ac-17827a4ac2dd" containerID="fc4f4b7e012d831333391208e1f2fc6c2e464b788888cc2e0a44a7cd85ab88ba" exitCode=0
Sep 18 13:44:47.955106 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.954943    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"
Sep 18 13:44:47.955106 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.954981    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l" event={"ID":"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd","Type":"ContainerDied","Data":"fc4f4b7e012d831333391208e1f2fc6c2e464b788888cc2e0a44a7cd85ab88ba"}
Sep 18 13:44:47.955106 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.955041    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l" event={"ID":"a9244fde-4e58-4ee0-a4ac-17827a4ac2dd","Type":"ContainerDied","Data":"83bfc346312fadf9e9020b3b04b8a2a78762cc8c46bb38e93916a6b46f84f08e"}
Sep 18 13:44:47.955106 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.955099    2566 scope.go:117] "RemoveContainer" containerID="fc4f4b7e012d831333391208e1f2fc6c2e464b788888cc2e0a44a7cd85ab88ba"
Sep 18 13:44:47.964816 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.964787    2566 scope.go:117] "RemoveContainer" containerID="33f3e61a0973b8510d06445884f51cdec4904fc66fb653ebe901dc0cf72830d6"
Sep 18 13:44:47.973131 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.973113    2566 scope.go:117] "RemoveContainer" containerID="fc4f4b7e012d831333391208e1f2fc6c2e464b788888cc2e0a44a7cd85ab88ba"
Sep 18 13:44:47.973370 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:44:47.973354    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4f4b7e012d831333391208e1f2fc6c2e464b788888cc2e0a44a7cd85ab88ba\": container with ID starting with fc4f4b7e012d831333391208e1f2fc6c2e464b788888cc2e0a44a7cd85ab88ba not found: ID does not exist" containerID="fc4f4b7e012d831333391208e1f2fc6c2e464b788888cc2e0a44a7cd85ab88ba"
Sep 18 13:44:47.973418 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.973377    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4f4b7e012d831333391208e1f2fc6c2e464b788888cc2e0a44a7cd85ab88ba"} err="failed to get container status \"fc4f4b7e012d831333391208e1f2fc6c2e464b788888cc2e0a44a7cd85ab88ba\": rpc error: code = NotFound desc = could not find container \"fc4f4b7e012d831333391208e1f2fc6c2e464b788888cc2e0a44a7cd85ab88ba\": container with ID starting with fc4f4b7e012d831333391208e1f2fc6c2e464b788888cc2e0a44a7cd85ab88ba not found: ID does not exist"
Sep 18 13:44:47.973418 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.973393    2566 scope.go:117] "RemoveContainer" containerID="33f3e61a0973b8510d06445884f51cdec4904fc66fb653ebe901dc0cf72830d6"
Sep 18 13:44:47.973623 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:44:47.973589    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33f3e61a0973b8510d06445884f51cdec4904fc66fb653ebe901dc0cf72830d6\": container with ID starting with 33f3e61a0973b8510d06445884f51cdec4904fc66fb653ebe901dc0cf72830d6 not found: ID does not exist" containerID="33f3e61a0973b8510d06445884f51cdec4904fc66fb653ebe901dc0cf72830d6"
Sep 18 13:44:47.973717 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.973626    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33f3e61a0973b8510d06445884f51cdec4904fc66fb653ebe901dc0cf72830d6"} err="failed to get container status \"33f3e61a0973b8510d06445884f51cdec4904fc66fb653ebe901dc0cf72830d6\": rpc error: code = NotFound desc = could not find container \"33f3e61a0973b8510d06445884f51cdec4904fc66fb653ebe901dc0cf72830d6\": container with ID starting with 33f3e61a0973b8510d06445884f51cdec4904fc66fb653ebe901dc0cf72830d6 not found: ID does not exist"
Sep 18 13:44:47.981412 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.981388    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"]
Sep 18 13:44:47.993665 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:47.993644    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-92b54f-openshift-default-df88d779b-qzb8l"]
Sep 18 13:44:48.077032 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:48.077001    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9244fde-4e58-4ee0-a4ac-17827a4ac2dd" path="/var/lib/kubelet/pods/a9244fde-4e58-4ee0-a4ac-17827a4ac2dd/volumes"
Sep 18 13:44:52.137952 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:52.137926    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k86vn_55e33e14-3f7a-4793-a724-9fd5022740d7/console-operator/1.log"
Sep 18 13:44:52.141030 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:44:52.141003    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k86vn_55e33e14-3f7a-4793-a724-9fd5022740d7/console-operator/1.log"
Sep 18 13:45:00.158160 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.158122    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828985-5gmqn"]
Sep 18 13:45:00.158731 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.158711    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17414e82-b0f6-4822-9eda-0b7d0a567925" containerName="istio-proxy"
Sep 18 13:45:00.158837 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.158733    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="17414e82-b0f6-4822-9eda-0b7d0a567925" containerName="istio-proxy"
Sep 18 13:45:00.158837 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.158759    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9244fde-4e58-4ee0-a4ac-17827a4ac2dd" containerName="istio-proxy"
Sep 18 13:45:00.158837 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.158767    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9244fde-4e58-4ee0-a4ac-17827a4ac2dd" containerName="istio-proxy"
Sep 18 13:45:00.158837 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.158779    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17414e82-b0f6-4822-9eda-0b7d0a567925" containerName="istio-proxy"
Sep 18 13:45:00.158837 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.158786    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="17414e82-b0f6-4822-9eda-0b7d0a567925" containerName="istio-proxy"
Sep 18 13:45:00.158837 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.158801    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a8735a2-d52e-4781-bff2-6b516ecf5cec" containerName="main"
Sep 18 13:45:00.158837 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.158809    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a8735a2-d52e-4781-bff2-6b516ecf5cec" containerName="main"
Sep 18 13:45:00.158837 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.158839    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50fc70eb-512e-42d0-89ef-f91eb7e00e99" containerName="main"
Sep 18 13:45:00.159234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.158847    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="50fc70eb-512e-42d0-89ef-f91eb7e00e99" containerName="main"
Sep 18 13:45:00.159234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.158856    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6ad330a-f565-4baf-827e-535cb5864563" containerName="main"
Sep 18 13:45:00.159234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.158864    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6ad330a-f565-4baf-827e-535cb5864563" containerName="main"
Sep 18 13:45:00.159234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.158876    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a1a326-b1b7-4266-9907-d729cabc4d58" containerName="istio-proxy"
Sep 18 13:45:00.159234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.158885    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a1a326-b1b7-4266-9907-d729cabc4d58" containerName="istio-proxy"
Sep 18 13:45:00.159234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.158894    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa415372-2fcd-4394-9976-f73d871b2cad" containerName="revoke-keys"
Sep 18 13:45:00.159234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.158902    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa415372-2fcd-4394-9976-f73d871b2cad" containerName="revoke-keys"
Sep 18 13:45:00.159234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.158915    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a1a326-b1b7-4266-9907-d729cabc4d58" containerName="istio-proxy"
Sep 18 13:45:00.159234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.158924    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a1a326-b1b7-4266-9907-d729cabc4d58" containerName="istio-proxy"
Sep 18 13:45:00.159234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.159025    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="17414e82-b0f6-4822-9eda-0b7d0a567925" containerName="istio-proxy"
Sep 18 13:45:00.159234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.159037    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9244fde-4e58-4ee0-a4ac-17827a4ac2dd" containerName="istio-proxy"
Sep 18 13:45:00.159234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.159049    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa415372-2fcd-4394-9976-f73d871b2cad" containerName="revoke-keys"
Sep 18 13:45:00.159234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.159059    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9244fde-4e58-4ee0-a4ac-17827a4ac2dd" containerName="istio-proxy"
Sep 18 13:45:00.159234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.159071    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="26a1a326-b1b7-4266-9907-d729cabc4d58" containerName="istio-proxy"
Sep 18 13:45:00.159234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.159081    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="17414e82-b0f6-4822-9eda-0b7d0a567925" containerName="istio-proxy"
Sep 18 13:45:00.159234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.159090    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="50fc70eb-512e-42d0-89ef-f91eb7e00e99" containerName="main"
Sep 18 13:45:00.159234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.159099    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="26a1a326-b1b7-4266-9907-d729cabc4d58" containerName="istio-proxy"
Sep 18 13:45:00.159234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.159107    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6ad330a-f565-4baf-827e-535cb5864563" containerName="main"
Sep 18 13:45:00.159234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.159114    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a8735a2-d52e-4781-bff2-6b516ecf5cec" containerName="main"
Sep 18 13:45:00.162361 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.162338    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828985-5gmqn"
Sep 18 13:45:00.164864 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.164838    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nh2cd\""
Sep 18 13:45:00.166703 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.166681    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c50e10-29828985-6snkz"]
Sep 18 13:45:00.167083 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.167066    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9244fde-4e58-4ee0-a4ac-17827a4ac2dd" containerName="istio-proxy"
Sep 18 13:45:00.167165 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.167085    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9244fde-4e58-4ee0-a4ac-17827a4ac2dd" containerName="istio-proxy"
Sep 18 13:45:00.170114 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.170064    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c50e10-29828985-6snkz"
Sep 18 13:45:00.173101 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.173084    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828985-5gmqn"
Sep 18 13:45:00.180987 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.180963    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828985-5gmqn"]
Sep 18 13:45:00.181681 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.181664    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c50e10-29828985-6snkz"
Sep 18 13:45:00.197223 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.197140    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c50e10-29828985-6snkz"]
Sep 18 13:45:00.464500 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.464466    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828985-5gmqn"]
Sep 18 13:45:00.467577 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:45:00.467546    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3221eb85_be0e_4232_a759_215335aae2b1.slice/crio-247295708e5c3c5fb993b3bfccabb397a866a1792432c77188856681deffce5b WatchSource:0}: Error finding container 247295708e5c3c5fb993b3bfccabb397a866a1792432c77188856681deffce5b: Status 404 returned error can't find the container with id 247295708e5c3c5fb993b3bfccabb397a866a1792432c77188856681deffce5b
Sep 18 13:45:00.474349 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:00.474323    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c50e10-29828985-6snkz"]
Sep 18 13:45:00.476924 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:45:00.476894    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c43d049_12b7_4c6f_ba03_d7b3895a9e31.slice/crio-8a439afdc1fc130b75891d62735cc66c7e8409e85cfeaf38e03fcdb3ef6e3ab0 WatchSource:0}: Error finding container 8a439afdc1fc130b75891d62735cc66c7e8409e85cfeaf38e03fcdb3ef6e3ab0: Status 404 returned error can't find the container with id 8a439afdc1fc130b75891d62735cc66c7e8409e85cfeaf38e03fcdb3ef6e3ab0
Sep 18 13:45:01.022215 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:01.022111    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c50e10-29828985-6snkz" event={"ID":"2c43d049-12b7-4c6f-ba03-d7b3895a9e31","Type":"ContainerStarted","Data":"8515657b47b25c1ba428a20da4e96cb27dcc5d9f3fd2c961fbbf122adb7c7c9f"}
Sep 18 13:45:01.022215 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:01.022172    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c50e10-29828985-6snkz" event={"ID":"2c43d049-12b7-4c6f-ba03-d7b3895a9e31","Type":"ContainerStarted","Data":"8a439afdc1fc130b75891d62735cc66c7e8409e85cfeaf38e03fcdb3ef6e3ab0"}
Sep 18 13:45:01.023842 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:01.023814    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828985-5gmqn" event={"ID":"3221eb85-be0e-4232-a759-215335aae2b1","Type":"ContainerStarted","Data":"3aa4489c06eef816b16c9b481e76639238388459a806188831c5e8cd3f665643"}
Sep 18 13:45:01.023966 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:01.023854    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828985-5gmqn" event={"ID":"3221eb85-be0e-4232-a759-215335aae2b1","Type":"ContainerStarted","Data":"247295708e5c3c5fb993b3bfccabb397a866a1792432c77188856681deffce5b"}
Sep 18 13:45:01.036794 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:01.036731    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c50e10-29828985-6snkz" podStartSLOduration=1.036712955 podStartE2EDuration="1.036712955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:01.036521148 +0000 UTC m=+1809.471397563" watchObservedRunningTime="2026-09-18 13:45:01.036712955 +0000 UTC m=+1809.471589358"
Sep 18 13:45:01.054097 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:01.054034    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828985-5gmqn" podStartSLOduration=1.054010527 podStartE2EDuration="1.054010527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:01.052702355 +0000 UTC m=+1809.487578759" watchObservedRunningTime="2026-09-18 13:45:01.054010527 +0000 UTC m=+1809.488886932"
Sep 18 13:45:02.031787 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:02.031749    2566 generic.go:356] "Generic (PLEG): container finished" podID="3221eb85-be0e-4232-a759-215335aae2b1" containerID="3aa4489c06eef816b16c9b481e76639238388459a806188831c5e8cd3f665643" exitCode=0
Sep 18 13:45:02.032286 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:02.031839    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828985-5gmqn" event={"ID":"3221eb85-be0e-4232-a759-215335aae2b1","Type":"ContainerDied","Data":"3aa4489c06eef816b16c9b481e76639238388459a806188831c5e8cd3f665643"}
Sep 18 13:45:02.033456 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:02.033431    2566 generic.go:356] "Generic (PLEG): container finished" podID="2c43d049-12b7-4c6f-ba03-d7b3895a9e31" containerID="8515657b47b25c1ba428a20da4e96cb27dcc5d9f3fd2c961fbbf122adb7c7c9f" exitCode=0
Sep 18 13:45:02.033572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:02.033468    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c50e10-29828985-6snkz" event={"ID":"2c43d049-12b7-4c6f-ba03-d7b3895a9e31","Type":"ContainerDied","Data":"8515657b47b25c1ba428a20da4e96cb27dcc5d9f3fd2c961fbbf122adb7c7c9f"}
Sep 18 13:45:03.198745 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:03.198720    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828985-5gmqn"
Sep 18 13:45:03.201747 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:03.201728    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c50e10-29828985-6snkz"
Sep 18 13:45:04.045947 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:04.045914    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c50e10-29828985-6snkz"
Sep 18 13:45:04.046153 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:04.045918    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c50e10-29828985-6snkz" event={"ID":"2c43d049-12b7-4c6f-ba03-d7b3895a9e31","Type":"ContainerDied","Data":"8a439afdc1fc130b75891d62735cc66c7e8409e85cfeaf38e03fcdb3ef6e3ab0"}
Sep 18 13:45:04.046153 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:04.046032    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a439afdc1fc130b75891d62735cc66c7e8409e85cfeaf38e03fcdb3ef6e3ab0"
Sep 18 13:45:04.047527 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:04.047502    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828985-5gmqn"
Sep 18 13:45:04.047664 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:04.047544    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828985-5gmqn" event={"ID":"3221eb85-be0e-4232-a759-215335aae2b1","Type":"ContainerDied","Data":"247295708e5c3c5fb993b3bfccabb397a866a1792432c77188856681deffce5b"}
Sep 18 13:45:04.047664 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:04.047610    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="247295708e5c3c5fb993b3bfccabb397a866a1792432c77188856681deffce5b"
Sep 18 13:45:04.405174 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:04.405131    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"]
Sep 18 13:45:04.421936 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:04.421905    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"]
Sep 18 13:45:15.284253 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:15.284213    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828985-5gmqn"]
Sep 18 13:45:15.286205 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:15.286179    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828970-xnrhx"]
Sep 18 13:45:15.290460 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:15.290429    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828970-xnrhx"]
Sep 18 13:45:15.295848 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:15.295822    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5be8a3-29828985-5gmqn"]
Sep 18 13:45:15.365218 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:15.365180    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn"]
Sep 18 13:45:15.365468 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:15.365437    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn" podUID="7fd8a897-c4ff-4183-8ab5-412d5a406847" containerName="payload-pre-processing" containerID="cri-o://cdd158261e83ae82e497885f7e6a672f1b0243e8193f119493fd394948c56b64" gracePeriod=30
Sep 18 13:45:15.631708 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:15.631680    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn"
Sep 18 13:45:15.701730 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:15.701695    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vkdw\" (UniqueName: \"kubernetes.io/projected/7fd8a897-c4ff-4183-8ab5-412d5a406847-kube-api-access-9vkdw\") pod \"7fd8a897-c4ff-4183-8ab5-412d5a406847\" (UID: \"7fd8a897-c4ff-4183-8ab5-412d5a406847\") "
Sep 18 13:45:15.701730 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:15.701735    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fd8a897-c4ff-4183-8ab5-412d5a406847-plugins-config-volume\") pod \"7fd8a897-c4ff-4183-8ab5-412d5a406847\" (UID: \"7fd8a897-c4ff-4183-8ab5-412d5a406847\") "
Sep 18 13:45:15.702222 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:15.702188    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fd8a897-c4ff-4183-8ab5-412d5a406847-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7fd8a897-c4ff-4183-8ab5-412d5a406847" (UID: "7fd8a897-c4ff-4183-8ab5-412d5a406847"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:45:15.703946 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:15.703916    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd8a897-c4ff-4183-8ab5-412d5a406847-kube-api-access-9vkdw" (OuterVolumeSpecName: "kube-api-access-9vkdw") pod "7fd8a897-c4ff-4183-8ab5-412d5a406847" (UID: "7fd8a897-c4ff-4183-8ab5-412d5a406847"). InnerVolumeSpecName "kube-api-access-9vkdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:45:15.803210 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:15.803119    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vkdw\" (UniqueName: \"kubernetes.io/projected/7fd8a897-c4ff-4183-8ab5-412d5a406847-kube-api-access-9vkdw\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:45:15.803210 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:15.803153    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fd8a897-c4ff-4183-8ab5-412d5a406847-plugins-config-volume\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:45:16.077547 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:16.077459    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1370e08f-2628-4be3-90ac-4568d17d1bb3" path="/var/lib/kubelet/pods/1370e08f-2628-4be3-90ac-4568d17d1bb3/volumes"
Sep 18 13:45:16.077752 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:16.077739    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3221eb85-be0e-4232-a759-215335aae2b1" path="/var/lib/kubelet/pods/3221eb85-be0e-4232-a759-215335aae2b1/volumes"
Sep 18 13:45:16.113720 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:16.113680    2566 generic.go:356] "Generic (PLEG): container finished" podID="7fd8a897-c4ff-4183-8ab5-412d5a406847" containerID="cdd158261e83ae82e497885f7e6a672f1b0243e8193f119493fd394948c56b64" exitCode=0
Sep 18 13:45:16.113934 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:16.113732    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn"
Sep 18 13:45:16.113934 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:16.113780    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn" event={"ID":"7fd8a897-c4ff-4183-8ab5-412d5a406847","Type":"ContainerDied","Data":"cdd158261e83ae82e497885f7e6a672f1b0243e8193f119493fd394948c56b64"}
Sep 18 13:45:16.113934 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:16.113844    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn" event={"ID":"7fd8a897-c4ff-4183-8ab5-412d5a406847","Type":"ContainerDied","Data":"8f10457b627c191ea2db32f0d3c1f17cbc06d716b2de7590647833f7977a247e"}
Sep 18 13:45:16.113934 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:16.113874    2566 scope.go:117] "RemoveContainer" containerID="cdd158261e83ae82e497885f7e6a672f1b0243e8193f119493fd394948c56b64"
Sep 18 13:45:16.123173 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:16.123153    2566 scope.go:117] "RemoveContainer" containerID="cdd158261e83ae82e497885f7e6a672f1b0243e8193f119493fd394948c56b64"
Sep 18 13:45:16.123502 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:45:16.123475    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd158261e83ae82e497885f7e6a672f1b0243e8193f119493fd394948c56b64\": container with ID starting with cdd158261e83ae82e497885f7e6a672f1b0243e8193f119493fd394948c56b64 not found: ID does not exist" containerID="cdd158261e83ae82e497885f7e6a672f1b0243e8193f119493fd394948c56b64"
Sep 18 13:45:16.123623 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:16.123508    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd158261e83ae82e497885f7e6a672f1b0243e8193f119493fd394948c56b64"} err="failed to get container status \"cdd158261e83ae82e497885f7e6a672f1b0243e8193f119493fd394948c56b64\": rpc error: code = NotFound desc = could not find container \"cdd158261e83ae82e497885f7e6a672f1b0243e8193f119493fd394948c56b64\": container with ID starting with cdd158261e83ae82e497885f7e6a672f1b0243e8193f119493fd394948c56b64 not found: ID does not exist"
Sep 18 13:45:16.141408 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:16.141376    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn"]
Sep 18 13:45:16.152173 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:16.152139    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5be8a3-65bfc65c74-m2lrn"]
Sep 18 13:45:18.077295 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:18.077261    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fd8a897-c4ff-4183-8ab5-412d5a406847" path="/var/lib/kubelet/pods/7fd8a897-c4ff-4183-8ab5-412d5a406847/volumes"
Sep 18 13:45:19.424390 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.424349    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj" podUID="80f3f049-86e4-40ba-959f-be09abb3138f" containerName="main" containerID="cri-o://136eaec3eb4bc684b87bd71cb35707515ac81548cd90e03c5e5cdf4226bc13e7" gracePeriod=45
Sep 18 13:45:19.440553 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.440519    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w" podUID="16906812-0e34-470c-b4bd-30f5e61d3f9b" containerName="main" containerID="cri-o://1b3872acaabe5f18b9dc8f4b9aed5fc68321b5fe3a616fe7136a9a4cb237a007" gracePeriod=45
Sep 18 13:45:19.720653 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.720621    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:45:19.723455 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.723432    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:45:19.838248 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.838209    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f3f049-86e4-40ba-959f-be09abb3138f-tls-certs\") pod \"80f3f049-86e4-40ba-959f-be09abb3138f\" (UID: \"80f3f049-86e4-40ba-959f-be09abb3138f\") "
Sep 18 13:45:19.838410 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.838299    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f3f049-86e4-40ba-959f-be09abb3138f-dshm\") pod \"80f3f049-86e4-40ba-959f-be09abb3138f\" (UID: \"80f3f049-86e4-40ba-959f-be09abb3138f\") "
Sep 18 13:45:19.838410 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.838392    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16906812-0e34-470c-b4bd-30f5e61d3f9b-model-cache\") pod \"16906812-0e34-470c-b4bd-30f5e61d3f9b\" (UID: \"16906812-0e34-470c-b4bd-30f5e61d3f9b\") "
Sep 18 13:45:19.838508 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.838449    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16906812-0e34-470c-b4bd-30f5e61d3f9b-tls-certs\") pod \"16906812-0e34-470c-b4bd-30f5e61d3f9b\" (UID: \"16906812-0e34-470c-b4bd-30f5e61d3f9b\") "
Sep 18 13:45:19.838508 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.838481    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16906812-0e34-470c-b4bd-30f5e61d3f9b-tmp-dir\") pod \"16906812-0e34-470c-b4bd-30f5e61d3f9b\" (UID: \"16906812-0e34-470c-b4bd-30f5e61d3f9b\") "
Sep 18 13:45:19.838620 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.838509    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16906812-0e34-470c-b4bd-30f5e61d3f9b-home\") pod \"16906812-0e34-470c-b4bd-30f5e61d3f9b\" (UID: \"16906812-0e34-470c-b4bd-30f5e61d3f9b\") "
Sep 18 13:45:19.838620 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.838536    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f3f049-86e4-40ba-959f-be09abb3138f-home\") pod \"80f3f049-86e4-40ba-959f-be09abb3138f\" (UID: \"80f3f049-86e4-40ba-959f-be09abb3138f\") "
Sep 18 13:45:19.838620 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.838572    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zh9zj\" (UniqueName: \"kubernetes.io/projected/16906812-0e34-470c-b4bd-30f5e61d3f9b-kube-api-access-zh9zj\") pod \"16906812-0e34-470c-b4bd-30f5e61d3f9b\" (UID: \"16906812-0e34-470c-b4bd-30f5e61d3f9b\") "
Sep 18 13:45:19.838798 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.838647    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cm7vw\" (UniqueName: \"kubernetes.io/projected/80f3f049-86e4-40ba-959f-be09abb3138f-kube-api-access-cm7vw\") pod \"80f3f049-86e4-40ba-959f-be09abb3138f\" (UID: \"80f3f049-86e4-40ba-959f-be09abb3138f\") "
Sep 18 13:45:19.838798 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.838678    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f3f049-86e4-40ba-959f-be09abb3138f-tmp-dir\") pod \"80f3f049-86e4-40ba-959f-be09abb3138f\" (UID: \"80f3f049-86e4-40ba-959f-be09abb3138f\") "
Sep 18 13:45:19.838798 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.838730    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f3f049-86e4-40ba-959f-be09abb3138f-model-cache\") pod \"80f3f049-86e4-40ba-959f-be09abb3138f\" (UID: \"80f3f049-86e4-40ba-959f-be09abb3138f\") "
Sep 18 13:45:19.838962 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.838798    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16906812-0e34-470c-b4bd-30f5e61d3f9b-dshm\") pod \"16906812-0e34-470c-b4bd-30f5e61d3f9b\" (UID: \"16906812-0e34-470c-b4bd-30f5e61d3f9b\") "
Sep 18 13:45:19.839058 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.838638    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16906812-0e34-470c-b4bd-30f5e61d3f9b-model-cache" (OuterVolumeSpecName: "model-cache") pod "16906812-0e34-470c-b4bd-30f5e61d3f9b" (UID: "16906812-0e34-470c-b4bd-30f5e61d3f9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:45:19.839637 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.839154    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16906812-0e34-470c-b4bd-30f5e61d3f9b-model-cache\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:45:19.839637 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.838838    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16906812-0e34-470c-b4bd-30f5e61d3f9b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "16906812-0e34-470c-b4bd-30f5e61d3f9b" (UID: "16906812-0e34-470c-b4bd-30f5e61d3f9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:45:19.839637 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.839019    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f3f049-86e4-40ba-959f-be09abb3138f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "80f3f049-86e4-40ba-959f-be09abb3138f" (UID: "80f3f049-86e4-40ba-959f-be09abb3138f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:45:19.839637 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.839393    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f3f049-86e4-40ba-959f-be09abb3138f-model-cache" (OuterVolumeSpecName: "model-cache") pod "80f3f049-86e4-40ba-959f-be09abb3138f" (UID: "80f3f049-86e4-40ba-959f-be09abb3138f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:45:19.839637 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.839558    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f3f049-86e4-40ba-959f-be09abb3138f-home" (OuterVolumeSpecName: "home") pod "80f3f049-86e4-40ba-959f-be09abb3138f" (UID: "80f3f049-86e4-40ba-959f-be09abb3138f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:45:19.840191 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.840155    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16906812-0e34-470c-b4bd-30f5e61d3f9b-home" (OuterVolumeSpecName: "home") pod "16906812-0e34-470c-b4bd-30f5e61d3f9b" (UID: "16906812-0e34-470c-b4bd-30f5e61d3f9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:45:19.841058 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.841029    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80f3f049-86e4-40ba-959f-be09abb3138f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "80f3f049-86e4-40ba-959f-be09abb3138f" (UID: "80f3f049-86e4-40ba-959f-be09abb3138f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:45:19.841322 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.841292    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f3f049-86e4-40ba-959f-be09abb3138f-dshm" (OuterVolumeSpecName: "dshm") pod "80f3f049-86e4-40ba-959f-be09abb3138f" (UID: "80f3f049-86e4-40ba-959f-be09abb3138f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:45:19.841430 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.841335    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16906812-0e34-470c-b4bd-30f5e61d3f9b-kube-api-access-zh9zj" (OuterVolumeSpecName: "kube-api-access-zh9zj") pod "16906812-0e34-470c-b4bd-30f5e61d3f9b" (UID: "16906812-0e34-470c-b4bd-30f5e61d3f9b"). InnerVolumeSpecName "kube-api-access-zh9zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:45:19.841430 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.841354    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f3f049-86e4-40ba-959f-be09abb3138f-kube-api-access-cm7vw" (OuterVolumeSpecName: "kube-api-access-cm7vw") pod "80f3f049-86e4-40ba-959f-be09abb3138f" (UID: "80f3f049-86e4-40ba-959f-be09abb3138f"). InnerVolumeSpecName "kube-api-access-cm7vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:45:19.841552 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.841448    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16906812-0e34-470c-b4bd-30f5e61d3f9b-dshm" (OuterVolumeSpecName: "dshm") pod "16906812-0e34-470c-b4bd-30f5e61d3f9b" (UID: "16906812-0e34-470c-b4bd-30f5e61d3f9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:45:19.841552 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.841494    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16906812-0e34-470c-b4bd-30f5e61d3f9b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "16906812-0e34-470c-b4bd-30f5e61d3f9b" (UID: "16906812-0e34-470c-b4bd-30f5e61d3f9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:45:19.940534 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.940451    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm7vw\" (UniqueName: \"kubernetes.io/projected/80f3f049-86e4-40ba-959f-be09abb3138f-kube-api-access-cm7vw\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:45:19.940534 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.940504    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f3f049-86e4-40ba-959f-be09abb3138f-tmp-dir\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:45:19.940534 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.940516    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f3f049-86e4-40ba-959f-be09abb3138f-model-cache\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:45:19.940534 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.940524    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16906812-0e34-470c-b4bd-30f5e61d3f9b-dshm\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:45:19.940534 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.940532    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f3f049-86e4-40ba-959f-be09abb3138f-tls-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:45:19.940534 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.940540    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f3f049-86e4-40ba-959f-be09abb3138f-dshm\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:45:19.940863 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.940547    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16906812-0e34-470c-b4bd-30f5e61d3f9b-tls-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:45:19.940863 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.940554    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16906812-0e34-470c-b4bd-30f5e61d3f9b-tmp-dir\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:45:19.940863 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.940561    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16906812-0e34-470c-b4bd-30f5e61d3f9b-home\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:45:19.940863 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.940568    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f3f049-86e4-40ba-959f-be09abb3138f-home\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:45:19.940863 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:19.940575    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh9zj\" (UniqueName: \"kubernetes.io/projected/16906812-0e34-470c-b4bd-30f5e61d3f9b-kube-api-access-zh9zj\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:45:20.137174 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:20.137140    2566 generic.go:356] "Generic (PLEG): container finished" podID="16906812-0e34-470c-b4bd-30f5e61d3f9b" containerID="1b3872acaabe5f18b9dc8f4b9aed5fc68321b5fe3a616fe7136a9a4cb237a007" exitCode=0
Sep 18 13:45:20.137350 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:20.137181    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"
Sep 18 13:45:20.137350 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:20.137215    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w" event={"ID":"16906812-0e34-470c-b4bd-30f5e61d3f9b","Type":"ContainerDied","Data":"1b3872acaabe5f18b9dc8f4b9aed5fc68321b5fe3a616fe7136a9a4cb237a007"}
Sep 18 13:45:20.137350 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:20.137267    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w" event={"ID":"16906812-0e34-470c-b4bd-30f5e61d3f9b","Type":"ContainerDied","Data":"dd44a683aa5128f9cdc265607ccf8bf33f94f50ace02b6b2606b2641b7085405"}
Sep 18 13:45:20.137350 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:20.137283    2566 scope.go:117] "RemoveContainer" containerID="1b3872acaabe5f18b9dc8f4b9aed5fc68321b5fe3a616fe7136a9a4cb237a007"
Sep 18 13:45:20.138765 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:20.138746    2566 generic.go:356] "Generic (PLEG): container finished" podID="80f3f049-86e4-40ba-959f-be09abb3138f" containerID="136eaec3eb4bc684b87bd71cb35707515ac81548cd90e03c5e5cdf4226bc13e7" exitCode=0
Sep 18 13:45:20.138873 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:20.138775    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj" event={"ID":"80f3f049-86e4-40ba-959f-be09abb3138f","Type":"ContainerDied","Data":"136eaec3eb4bc684b87bd71cb35707515ac81548cd90e03c5e5cdf4226bc13e7"}
Sep 18 13:45:20.138873 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:20.138788    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"
Sep 18 13:45:20.138873 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:20.138801    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj" event={"ID":"80f3f049-86e4-40ba-959f-be09abb3138f","Type":"ContainerDied","Data":"7d0c75a7a531f9fdb6bf8dbb6e454a15121f4c5323081fd1ed051e3d4b65826e"}
Sep 18 13:45:20.149145 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:20.149121    2566 scope.go:117] "RemoveContainer" containerID="1b3872acaabe5f18b9dc8f4b9aed5fc68321b5fe3a616fe7136a9a4cb237a007"
Sep 18 13:45:20.149492 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:45:20.149470    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3872acaabe5f18b9dc8f4b9aed5fc68321b5fe3a616fe7136a9a4cb237a007\": container with ID starting with 1b3872acaabe5f18b9dc8f4b9aed5fc68321b5fe3a616fe7136a9a4cb237a007 not found: ID does not exist" containerID="1b3872acaabe5f18b9dc8f4b9aed5fc68321b5fe3a616fe7136a9a4cb237a007"
Sep 18 13:45:20.149571 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:20.149500    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3872acaabe5f18b9dc8f4b9aed5fc68321b5fe3a616fe7136a9a4cb237a007"} err="failed to get container status \"1b3872acaabe5f18b9dc8f4b9aed5fc68321b5fe3a616fe7136a9a4cb237a007\": rpc error: code = NotFound desc = could not find container \"1b3872acaabe5f18b9dc8f4b9aed5fc68321b5fe3a616fe7136a9a4cb237a007\": container with ID starting with 1b3872acaabe5f18b9dc8f4b9aed5fc68321b5fe3a616fe7136a9a4cb237a007 not found: ID does not exist"
Sep 18 13:45:20.149571 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:20.149517    2566 scope.go:117] "RemoveContainer" containerID="136eaec3eb4bc684b87bd71cb35707515ac81548cd90e03c5e5cdf4226bc13e7"
Sep 18 13:45:20.157532 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:20.157511    2566 scope.go:117] "RemoveContainer" containerID="136eaec3eb4bc684b87bd71cb35707515ac81548cd90e03c5e5cdf4226bc13e7"
Sep 18 13:45:20.157865 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:45:20.157841    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"136eaec3eb4bc684b87bd71cb35707515ac81548cd90e03c5e5cdf4226bc13e7\": container with ID starting with 136eaec3eb4bc684b87bd71cb35707515ac81548cd90e03c5e5cdf4226bc13e7 not found: ID does not exist" containerID="136eaec3eb4bc684b87bd71cb35707515ac81548cd90e03c5e5cdf4226bc13e7"
Sep 18 13:45:20.157922 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:20.157877    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"136eaec3eb4bc684b87bd71cb35707515ac81548cd90e03c5e5cdf4226bc13e7"} err="failed to get container status \"136eaec3eb4bc684b87bd71cb35707515ac81548cd90e03c5e5cdf4226bc13e7\": rpc error: code = NotFound desc = could not find container \"136eaec3eb4bc684b87bd71cb35707515ac81548cd90e03c5e5cdf4226bc13e7\": container with ID starting with 136eaec3eb4bc684b87bd71cb35707515ac81548cd90e03c5e5cdf4226bc13e7 not found: ID does not exist"
Sep 18 13:45:20.177438 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:20.177377    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"]
Sep 18 13:45:20.179428 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:20.179403    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/distinct-w5-5be8a3-kserve-6bf65477fb-xgr6w"]
Sep 18 13:45:20.191818 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:20.191725    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"]
Sep 18 13:45:20.197362 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:20.197331    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj"]
Sep 18 13:45:22.076428 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:22.076386    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16906812-0e34-470c-b4bd-30f5e61d3f9b" path="/var/lib/kubelet/pods/16906812-0e34-470c-b4bd-30f5e61d3f9b/volumes"
Sep 18 13:45:22.077004 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:22.076981    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80f3f049-86e4-40ba-959f-be09abb3138f" path="/var/lib/kubelet/pods/80f3f049-86e4-40ba-959f-be09abb3138f/volumes"
Sep 18 13:45:22.794316 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:22.794278    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"]
Sep 18 13:45:24.607699 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:24.607654    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5be8a3/distinct-2-w5-5be8a3-kserve-75c7c795-msncj" podUID="80f3f049-86e4-40ba-959f-be09abb3138f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Sep 18 13:45:36.199846 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.199808    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"]
Sep 18 13:45:36.200360 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.200340    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3221eb85-be0e-4232-a759-215335aae2b1" containerName="cleanup"
Sep 18 13:45:36.200445 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.200364    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3221eb85-be0e-4232-a759-215335aae2b1" containerName="cleanup"
Sep 18 13:45:36.200445 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.200385    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c43d049-12b7-4c6f-ba03-d7b3895a9e31" containerName="cleanup"
Sep 18 13:45:36.200445 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.200394    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c43d049-12b7-4c6f-ba03-d7b3895a9e31" containerName="cleanup"
Sep 18 13:45:36.200445 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.200416    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16906812-0e34-470c-b4bd-30f5e61d3f9b" containerName="main"
Sep 18 13:45:36.200445 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.200425    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="16906812-0e34-470c-b4bd-30f5e61d3f9b" containerName="main"
Sep 18 13:45:36.200445 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.200442    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80f3f049-86e4-40ba-959f-be09abb3138f" containerName="main"
Sep 18 13:45:36.200781 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.200450    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="80f3f049-86e4-40ba-959f-be09abb3138f" containerName="main"
Sep 18 13:45:36.200781 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.200466    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd8a897-c4ff-4183-8ab5-412d5a406847" containerName="payload-pre-processing"
Sep 18 13:45:36.200781 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.200475    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd8a897-c4ff-4183-8ab5-412d5a406847" containerName="payload-pre-processing"
Sep 18 13:45:36.200781 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.200564    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fd8a897-c4ff-4183-8ab5-412d5a406847" containerName="payload-pre-processing"
Sep 18 13:45:36.200781 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.200576    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="16906812-0e34-470c-b4bd-30f5e61d3f9b" containerName="main"
Sep 18 13:45:36.200781 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.200586    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c43d049-12b7-4c6f-ba03-d7b3895a9e31" containerName="cleanup"
Sep 18 13:45:36.200781 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.200614    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3221eb85-be0e-4232-a759-215335aae2b1" containerName="cleanup"
Sep 18 13:45:36.200781 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.200622    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="80f3f049-86e4-40ba-959f-be09abb3138f" containerName="main"
Sep 18 13:45:36.203998 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.203975    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:45:36.208094 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.208072    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-537abe\"/\"default-dockercfg-vw4dd\""
Sep 18 13:45:36.208197 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.208080    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-537abe\"/\"sub-test-model-537abe-kserve-self-signed-certs\""
Sep 18 13:45:36.208607 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.208579    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-537abe\"/\"kube-root-ca.crt\""
Sep 18 13:45:36.208652 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.208637    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-537abe\"/\"openshift-service-ca.crt\""
Sep 18 13:45:36.217975 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.217947    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"]
Sep 18 13:45:36.389929 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.389894    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/710d35a1-6458-4638-b397-80c83f2c8afd-tmp-dir\") pod \"sub-test-model-537abe-kserve-85b869b7f7-bqgv9\" (UID: \"710d35a1-6458-4638-b397-80c83f2c8afd\") " pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:45:36.389929 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.389945    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/710d35a1-6458-4638-b397-80c83f2c8afd-home\") pod \"sub-test-model-537abe-kserve-85b869b7f7-bqgv9\" (UID: \"710d35a1-6458-4638-b397-80c83f2c8afd\") " pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:45:36.390195 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.390007    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/710d35a1-6458-4638-b397-80c83f2c8afd-dshm\") pod \"sub-test-model-537abe-kserve-85b869b7f7-bqgv9\" (UID: \"710d35a1-6458-4638-b397-80c83f2c8afd\") " pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:45:36.390195 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.390055    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/710d35a1-6458-4638-b397-80c83f2c8afd-tls-certs\") pod \"sub-test-model-537abe-kserve-85b869b7f7-bqgv9\" (UID: \"710d35a1-6458-4638-b397-80c83f2c8afd\") " pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:45:36.390195 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.390131    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/710d35a1-6458-4638-b397-80c83f2c8afd-model-cache\") pod \"sub-test-model-537abe-kserve-85b869b7f7-bqgv9\" (UID: \"710d35a1-6458-4638-b397-80c83f2c8afd\") " pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:45:36.390195 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.390164    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww7r9\" (UniqueName: \"kubernetes.io/projected/710d35a1-6458-4638-b397-80c83f2c8afd-kube-api-access-ww7r9\") pod \"sub-test-model-537abe-kserve-85b869b7f7-bqgv9\" (UID: \"710d35a1-6458-4638-b397-80c83f2c8afd\") " pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:45:36.491195 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.491103    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/710d35a1-6458-4638-b397-80c83f2c8afd-home\") pod \"sub-test-model-537abe-kserve-85b869b7f7-bqgv9\" (UID: \"710d35a1-6458-4638-b397-80c83f2c8afd\") " pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:45:36.491195 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.491159    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/710d35a1-6458-4638-b397-80c83f2c8afd-dshm\") pod \"sub-test-model-537abe-kserve-85b869b7f7-bqgv9\" (UID: \"710d35a1-6458-4638-b397-80c83f2c8afd\") " pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:45:36.491195 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.491182    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/710d35a1-6458-4638-b397-80c83f2c8afd-tls-certs\") pod \"sub-test-model-537abe-kserve-85b869b7f7-bqgv9\" (UID: \"710d35a1-6458-4638-b397-80c83f2c8afd\") " pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:45:36.491472 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.491228    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/710d35a1-6458-4638-b397-80c83f2c8afd-model-cache\") pod \"sub-test-model-537abe-kserve-85b869b7f7-bqgv9\" (UID: \"710d35a1-6458-4638-b397-80c83f2c8afd\") " pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:45:36.491472 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.491254    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww7r9\" (UniqueName: \"kubernetes.io/projected/710d35a1-6458-4638-b397-80c83f2c8afd-kube-api-access-ww7r9\") pod \"sub-test-model-537abe-kserve-85b869b7f7-bqgv9\" (UID: \"710d35a1-6458-4638-b397-80c83f2c8afd\") " pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:45:36.491472 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.491281    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/710d35a1-6458-4638-b397-80c83f2c8afd-tmp-dir\") pod \"sub-test-model-537abe-kserve-85b869b7f7-bqgv9\" (UID: \"710d35a1-6458-4638-b397-80c83f2c8afd\") " pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:45:36.491659 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.491522    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/710d35a1-6458-4638-b397-80c83f2c8afd-home\") pod \"sub-test-model-537abe-kserve-85b869b7f7-bqgv9\" (UID: \"710d35a1-6458-4638-b397-80c83f2c8afd\") " pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:45:36.491659 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.491570    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/710d35a1-6458-4638-b397-80c83f2c8afd-tmp-dir\") pod \"sub-test-model-537abe-kserve-85b869b7f7-bqgv9\" (UID: \"710d35a1-6458-4638-b397-80c83f2c8afd\") " pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:45:36.491740 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.491669    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/710d35a1-6458-4638-b397-80c83f2c8afd-model-cache\") pod \"sub-test-model-537abe-kserve-85b869b7f7-bqgv9\" (UID: \"710d35a1-6458-4638-b397-80c83f2c8afd\") " pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:45:36.493559 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.493539    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/710d35a1-6458-4638-b397-80c83f2c8afd-dshm\") pod \"sub-test-model-537abe-kserve-85b869b7f7-bqgv9\" (UID: \"710d35a1-6458-4638-b397-80c83f2c8afd\") " pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:45:36.493741 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.493723    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/710d35a1-6458-4638-b397-80c83f2c8afd-tls-certs\") pod \"sub-test-model-537abe-kserve-85b869b7f7-bqgv9\" (UID: \"710d35a1-6458-4638-b397-80c83f2c8afd\") " pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:45:36.499902 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.499881    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww7r9\" (UniqueName: \"kubernetes.io/projected/710d35a1-6458-4638-b397-80c83f2c8afd-kube-api-access-ww7r9\") pod \"sub-test-model-537abe-kserve-85b869b7f7-bqgv9\" (UID: \"710d35a1-6458-4638-b397-80c83f2c8afd\") " pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:45:36.513841 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.513819    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:45:36.789806 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:36.789779    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"]
Sep 18 13:45:36.792534 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:45:36.792508    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod710d35a1_6458_4638_b397_80c83f2c8afd.slice/crio-5b76ea59b1846838f7b7d376c7cadcc486c33fa1cce38b11e2fc6f36bc34a91a WatchSource:0}: Error finding container 5b76ea59b1846838f7b7d376c7cadcc486c33fa1cce38b11e2fc6f36bc34a91a: Status 404 returned error can't find the container with id 5b76ea59b1846838f7b7d376c7cadcc486c33fa1cce38b11e2fc6f36bc34a91a
Sep 18 13:45:37.232223 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:37.232190    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9" event={"ID":"710d35a1-6458-4638-b397-80c83f2c8afd","Type":"ContainerStarted","Data":"238fa99432547b616a4de1659269ce4eb2db3ae49f00ae2cb433d8b9d63d6ee2"}
Sep 18 13:45:37.232626 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:37.232232    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9" event={"ID":"710d35a1-6458-4638-b397-80c83f2c8afd","Type":"ContainerStarted","Data":"5b76ea59b1846838f7b7d376c7cadcc486c33fa1cce38b11e2fc6f36bc34a91a"}
Sep 18 13:45:37.250311 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:37.250265    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9" podStartSLOduration=1.250248692 podStartE2EDuration="1.250248692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:37.248087869 +0000 UTC m=+1845.682964275" watchObservedRunningTime="2026-09-18 13:45:37.250248692 +0000 UTC m=+1845.685125095"
Sep 18 13:45:37.815224 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:37.815180    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd" podUID="e66f5234-dc9b-4fc6-9d16-c520f1a08142" containerName="main" containerID="cri-o://dc534d2d58a80259e606e83b921497c3250d3ac7bc39c86a440d30b3abf6cf05" gracePeriod=45
Sep 18 13:45:38.068241 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.068178    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:45:38.210186 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.210147    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e66f5234-dc9b-4fc6-9d16-c520f1a08142-tls-certs\") pod \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\" (UID: \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\") "
Sep 18 13:45:38.210379 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.210208    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9clwf\" (UniqueName: \"kubernetes.io/projected/e66f5234-dc9b-4fc6-9d16-c520f1a08142-kube-api-access-9clwf\") pod \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\" (UID: \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\") "
Sep 18 13:45:38.210379 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.210267    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e66f5234-dc9b-4fc6-9d16-c520f1a08142-home\") pod \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\" (UID: \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\") "
Sep 18 13:45:38.210379 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.210283    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e66f5234-dc9b-4fc6-9d16-c520f1a08142-dshm\") pod \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\" (UID: \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\") "
Sep 18 13:45:38.210379 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.210299    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66f5234-dc9b-4fc6-9d16-c520f1a08142-model-cache\") pod \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\" (UID: \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\") "
Sep 18 13:45:38.210379 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.210361    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e66f5234-dc9b-4fc6-9d16-c520f1a08142-tmp-dir\") pod \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\" (UID: \"e66f5234-dc9b-4fc6-9d16-c520f1a08142\") "
Sep 18 13:45:38.210681 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.210627    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66f5234-dc9b-4fc6-9d16-c520f1a08142-home" (OuterVolumeSpecName: "home") pod "e66f5234-dc9b-4fc6-9d16-c520f1a08142" (UID: "e66f5234-dc9b-4fc6-9d16-c520f1a08142"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:45:38.210760 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.210657    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66f5234-dc9b-4fc6-9d16-c520f1a08142-model-cache" (OuterVolumeSpecName: "model-cache") pod "e66f5234-dc9b-4fc6-9d16-c520f1a08142" (UID: "e66f5234-dc9b-4fc6-9d16-c520f1a08142"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:45:38.210760 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.210714    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66f5234-dc9b-4fc6-9d16-c520f1a08142-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e66f5234-dc9b-4fc6-9d16-c520f1a08142" (UID: "e66f5234-dc9b-4fc6-9d16-c520f1a08142"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:45:38.212634 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.212607    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66f5234-dc9b-4fc6-9d16-c520f1a08142-kube-api-access-9clwf" (OuterVolumeSpecName: "kube-api-access-9clwf") pod "e66f5234-dc9b-4fc6-9d16-c520f1a08142" (UID: "e66f5234-dc9b-4fc6-9d16-c520f1a08142"). InnerVolumeSpecName "kube-api-access-9clwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:45:38.212723 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.212647    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e66f5234-dc9b-4fc6-9d16-c520f1a08142-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e66f5234-dc9b-4fc6-9d16-c520f1a08142" (UID: "e66f5234-dc9b-4fc6-9d16-c520f1a08142"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:45:38.212723 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.212656    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66f5234-dc9b-4fc6-9d16-c520f1a08142-dshm" (OuterVolumeSpecName: "dshm") pod "e66f5234-dc9b-4fc6-9d16-c520f1a08142" (UID: "e66f5234-dc9b-4fc6-9d16-c520f1a08142"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:45:38.238552 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.238518    2566 generic.go:356] "Generic (PLEG): container finished" podID="e66f5234-dc9b-4fc6-9d16-c520f1a08142" containerID="dc534d2d58a80259e606e83b921497c3250d3ac7bc39c86a440d30b3abf6cf05" exitCode=0
Sep 18 13:45:38.238960 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.238557    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"
Sep 18 13:45:38.238960 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.238609    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd" event={"ID":"e66f5234-dc9b-4fc6-9d16-c520f1a08142","Type":"ContainerDied","Data":"dc534d2d58a80259e606e83b921497c3250d3ac7bc39c86a440d30b3abf6cf05"}
Sep 18 13:45:38.238960 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.238653    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd" event={"ID":"e66f5234-dc9b-4fc6-9d16-c520f1a08142","Type":"ContainerDied","Data":"e919f9a39bc9bad0d448de384b01429112a856a6d4f59f3e5451e202d8ff0f46"}
Sep 18 13:45:38.238960 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.238671    2566 scope.go:117] "RemoveContainer" containerID="dc534d2d58a80259e606e83b921497c3250d3ac7bc39c86a440d30b3abf6cf05"
Sep 18 13:45:38.247446 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.247431    2566 scope.go:117] "RemoveContainer" containerID="dc534d2d58a80259e606e83b921497c3250d3ac7bc39c86a440d30b3abf6cf05"
Sep 18 13:45:38.247742 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:45:38.247721    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc534d2d58a80259e606e83b921497c3250d3ac7bc39c86a440d30b3abf6cf05\": container with ID starting with dc534d2d58a80259e606e83b921497c3250d3ac7bc39c86a440d30b3abf6cf05 not found: ID does not exist" containerID="dc534d2d58a80259e606e83b921497c3250d3ac7bc39c86a440d30b3abf6cf05"
Sep 18 13:45:38.247818 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.247755    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc534d2d58a80259e606e83b921497c3250d3ac7bc39c86a440d30b3abf6cf05"} err="failed to get container status \"dc534d2d58a80259e606e83b921497c3250d3ac7bc39c86a440d30b3abf6cf05\": rpc error: code = NotFound desc = could not find container \"dc534d2d58a80259e606e83b921497c3250d3ac7bc39c86a440d30b3abf6cf05\": container with ID starting with dc534d2d58a80259e606e83b921497c3250d3ac7bc39c86a440d30b3abf6cf05 not found: ID does not exist"
Sep 18 13:45:38.260877 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.260846    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"]
Sep 18 13:45:38.265290 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.265262    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c50e10/auth-test-model-c50e10-kserve-6df69ff995-f8gvd"]
Sep 18 13:45:38.311027 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.310989    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e66f5234-dc9b-4fc6-9d16-c520f1a08142-tmp-dir\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:45:38.311027 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.311017    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e66f5234-dc9b-4fc6-9d16-c520f1a08142-tls-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:45:38.311027 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.311027    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9clwf\" (UniqueName: \"kubernetes.io/projected/e66f5234-dc9b-4fc6-9d16-c520f1a08142-kube-api-access-9clwf\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:45:38.311027 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.311037    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e66f5234-dc9b-4fc6-9d16-c520f1a08142-home\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:45:38.311386 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.311046    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e66f5234-dc9b-4fc6-9d16-c520f1a08142-dshm\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:45:38.311386 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:38.311054    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66f5234-dc9b-4fc6-9d16-c520f1a08142-model-cache\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:45:40.076841 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:40.076798    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e66f5234-dc9b-4fc6-9d16-c520f1a08142" path="/var/lib/kubelet/pods/e66f5234-dc9b-4fc6-9d16-c520f1a08142/volumes"
Sep 18 13:45:46.514644 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:46.514575    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:45:46.514644 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:46.514656    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:45:46.527213 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:46.527183    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:45:47.297896 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:47.297866    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:45:52.430810 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:52.430780    2566 scope.go:117] "RemoveContainer" containerID="5e032634f5925d03b4ac978af1cc5b9a4debac1bbdc4c0d8a33787ba88873426"
Sep 18 13:45:52.439490 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:45:52.439467    2566 scope.go:117] "RemoveContainer" containerID="56dd882afbf02ef63999761094326cf4f83f451cbc0189729ae7f568f2cd0302"
Sep 18 13:46:43.222568 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:43.222535    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"]
Sep 18 13:46:58.241627 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.241549    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9" podUID="710d35a1-6458-4638-b397-80c83f2c8afd" containerName="main" containerID="cri-o://238fa99432547b616a4de1659269ce4eb2db3ae49f00ae2cb433d8b9d63d6ee2" gracePeriod=45
Sep 18 13:46:58.494114 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.494052    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:46:58.650178 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.650147    2566 generic.go:356] "Generic (PLEG): container finished" podID="710d35a1-6458-4638-b397-80c83f2c8afd" containerID="238fa99432547b616a4de1659269ce4eb2db3ae49f00ae2cb433d8b9d63d6ee2" exitCode=0
Sep 18 13:46:58.650354 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.650195    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"
Sep 18 13:46:58.650354 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.650238    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9" event={"ID":"710d35a1-6458-4638-b397-80c83f2c8afd","Type":"ContainerDied","Data":"238fa99432547b616a4de1659269ce4eb2db3ae49f00ae2cb433d8b9d63d6ee2"}
Sep 18 13:46:58.650354 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.650286    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9" event={"ID":"710d35a1-6458-4638-b397-80c83f2c8afd","Type":"ContainerDied","Data":"5b76ea59b1846838f7b7d376c7cadcc486c33fa1cce38b11e2fc6f36bc34a91a"}
Sep 18 13:46:58.650354 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.650309    2566 scope.go:117] "RemoveContainer" containerID="238fa99432547b616a4de1659269ce4eb2db3ae49f00ae2cb433d8b9d63d6ee2"
Sep 18 13:46:58.658999 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.658981    2566 scope.go:117] "RemoveContainer" containerID="238fa99432547b616a4de1659269ce4eb2db3ae49f00ae2cb433d8b9d63d6ee2"
Sep 18 13:46:58.659269 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:46:58.659249    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"238fa99432547b616a4de1659269ce4eb2db3ae49f00ae2cb433d8b9d63d6ee2\": container with ID starting with 238fa99432547b616a4de1659269ce4eb2db3ae49f00ae2cb433d8b9d63d6ee2 not found: ID does not exist" containerID="238fa99432547b616a4de1659269ce4eb2db3ae49f00ae2cb433d8b9d63d6ee2"
Sep 18 13:46:58.659321 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.659280    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"238fa99432547b616a4de1659269ce4eb2db3ae49f00ae2cb433d8b9d63d6ee2"} err="failed to get container status \"238fa99432547b616a4de1659269ce4eb2db3ae49f00ae2cb433d8b9d63d6ee2\": rpc error: code = NotFound desc = could not find container \"238fa99432547b616a4de1659269ce4eb2db3ae49f00ae2cb433d8b9d63d6ee2\": container with ID starting with 238fa99432547b616a4de1659269ce4eb2db3ae49f00ae2cb433d8b9d63d6ee2 not found: ID does not exist"
Sep 18 13:46:58.660501 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.660481    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/710d35a1-6458-4638-b397-80c83f2c8afd-tmp-dir\") pod \"710d35a1-6458-4638-b397-80c83f2c8afd\" (UID: \"710d35a1-6458-4638-b397-80c83f2c8afd\") "
Sep 18 13:46:58.660620 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.660513    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/710d35a1-6458-4638-b397-80c83f2c8afd-dshm\") pod \"710d35a1-6458-4638-b397-80c83f2c8afd\" (UID: \"710d35a1-6458-4638-b397-80c83f2c8afd\") "
Sep 18 13:46:58.660620 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.660538    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/710d35a1-6458-4638-b397-80c83f2c8afd-home\") pod \"710d35a1-6458-4638-b397-80c83f2c8afd\" (UID: \"710d35a1-6458-4638-b397-80c83f2c8afd\") "
Sep 18 13:46:58.660620 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.660580    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/710d35a1-6458-4638-b397-80c83f2c8afd-model-cache\") pod \"710d35a1-6458-4638-b397-80c83f2c8afd\" (UID: \"710d35a1-6458-4638-b397-80c83f2c8afd\") "
Sep 18 13:46:58.660784 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.660689    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww7r9\" (UniqueName: \"kubernetes.io/projected/710d35a1-6458-4638-b397-80c83f2c8afd-kube-api-access-ww7r9\") pod \"710d35a1-6458-4638-b397-80c83f2c8afd\" (UID: \"710d35a1-6458-4638-b397-80c83f2c8afd\") "
Sep 18 13:46:58.660784 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.660765    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/710d35a1-6458-4638-b397-80c83f2c8afd-tls-certs\") pod \"710d35a1-6458-4638-b397-80c83f2c8afd\" (UID: \"710d35a1-6458-4638-b397-80c83f2c8afd\") "
Sep 18 13:46:58.660893 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.660795    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710d35a1-6458-4638-b397-80c83f2c8afd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "710d35a1-6458-4638-b397-80c83f2c8afd" (UID: "710d35a1-6458-4638-b397-80c83f2c8afd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:46:58.660893 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.660807    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710d35a1-6458-4638-b397-80c83f2c8afd-home" (OuterVolumeSpecName: "home") pod "710d35a1-6458-4638-b397-80c83f2c8afd" (UID: "710d35a1-6458-4638-b397-80c83f2c8afd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:46:58.661003 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.660909    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710d35a1-6458-4638-b397-80c83f2c8afd-model-cache" (OuterVolumeSpecName: "model-cache") pod "710d35a1-6458-4638-b397-80c83f2c8afd" (UID: "710d35a1-6458-4638-b397-80c83f2c8afd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:46:58.661197 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.661178    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/710d35a1-6458-4638-b397-80c83f2c8afd-tmp-dir\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:46:58.661254 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.661200    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/710d35a1-6458-4638-b397-80c83f2c8afd-home\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:46:58.661254 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.661214    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/710d35a1-6458-4638-b397-80c83f2c8afd-model-cache\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:46:58.662841 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.662812    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710d35a1-6458-4638-b397-80c83f2c8afd-dshm" (OuterVolumeSpecName: "dshm") pod "710d35a1-6458-4638-b397-80c83f2c8afd" (UID: "710d35a1-6458-4638-b397-80c83f2c8afd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:46:58.662841 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.662835    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/710d35a1-6458-4638-b397-80c83f2c8afd-kube-api-access-ww7r9" (OuterVolumeSpecName: "kube-api-access-ww7r9") pod "710d35a1-6458-4638-b397-80c83f2c8afd" (UID: "710d35a1-6458-4638-b397-80c83f2c8afd"). InnerVolumeSpecName "kube-api-access-ww7r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:46:58.663006 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.662903    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/710d35a1-6458-4638-b397-80c83f2c8afd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "710d35a1-6458-4638-b397-80c83f2c8afd" (UID: "710d35a1-6458-4638-b397-80c83f2c8afd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:46:58.761959 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.761862    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww7r9\" (UniqueName: \"kubernetes.io/projected/710d35a1-6458-4638-b397-80c83f2c8afd-kube-api-access-ww7r9\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:46:58.761959 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.761895    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/710d35a1-6458-4638-b397-80c83f2c8afd-tls-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:46:58.761959 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.761905    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/710d35a1-6458-4638-b397-80c83f2c8afd-dshm\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:46:58.974916 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.974878    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"]
Sep 18 13:46:58.979779 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:46:58.979746    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-537abe/sub-test-model-537abe-kserve-85b869b7f7-bqgv9"]
Sep 18 13:47:00.077892 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:00.077853    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="710d35a1-6458-4638-b397-80c83f2c8afd" path="/var/lib/kubelet/pods/710d35a1-6458-4638-b397-80c83f2c8afd/volumes"
Sep 18 13:47:14.380791 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.380755    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"]
Sep 18 13:47:14.381232 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.381171    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e66f5234-dc9b-4fc6-9d16-c520f1a08142" containerName="main"
Sep 18 13:47:14.381232 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.381186    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e66f5234-dc9b-4fc6-9d16-c520f1a08142" containerName="main"
Sep 18 13:47:14.381232 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.381204    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="710d35a1-6458-4638-b397-80c83f2c8afd" containerName="main"
Sep 18 13:47:14.381232 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.381211    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="710d35a1-6458-4638-b397-80c83f2c8afd" containerName="main"
Sep 18 13:47:14.381369 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.381314    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e66f5234-dc9b-4fc6-9d16-c520f1a08142" containerName="main"
Sep 18 13:47:14.381369 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.381325    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="710d35a1-6458-4638-b397-80c83f2c8afd" containerName="main"
Sep 18 13:47:14.385690 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.385671    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:47:14.389747 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.389726    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c50e10\"/\"kube-root-ca.crt\""
Sep 18 13:47:14.389868 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.389728    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c50e10\"/\"default-dockercfg-g82ls\""
Sep 18 13:47:14.390650 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.390634    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c50e10\"/\"rate-test-model-c50e10-kserve-self-signed-certs\""
Sep 18 13:47:14.390726 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.390709    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c50e10\"/\"openshift-service-ca.crt\""
Sep 18 13:47:14.400068 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.400042    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"]
Sep 18 13:47:14.480128 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.480089    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-home\") pod \"rate-test-model-c50e10-kserve-777754bcc9-wxdq8\" (UID: \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:47:14.480324 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.480139    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-tmp-dir\") pod \"rate-test-model-c50e10-kserve-777754bcc9-wxdq8\" (UID: \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:47:14.480324 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.480164    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-dshm\") pod \"rate-test-model-c50e10-kserve-777754bcc9-wxdq8\" (UID: \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:47:14.480324 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.480206    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-model-cache\") pod \"rate-test-model-c50e10-kserve-777754bcc9-wxdq8\" (UID: \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:47:14.480324 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.480238    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-tls-certs\") pod \"rate-test-model-c50e10-kserve-777754bcc9-wxdq8\" (UID: \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:47:14.480324 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.480302    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f65lr\" (UniqueName: \"kubernetes.io/projected/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-kube-api-access-f65lr\") pod \"rate-test-model-c50e10-kserve-777754bcc9-wxdq8\" (UID: \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:47:14.580751 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.580712    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-home\") pod \"rate-test-model-c50e10-kserve-777754bcc9-wxdq8\" (UID: \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:47:14.580951 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.580761    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-tmp-dir\") pod \"rate-test-model-c50e10-kserve-777754bcc9-wxdq8\" (UID: \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:47:14.580951 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.580789    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-dshm\") pod \"rate-test-model-c50e10-kserve-777754bcc9-wxdq8\" (UID: \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:47:14.580951 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.580837    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-model-cache\") pod \"rate-test-model-c50e10-kserve-777754bcc9-wxdq8\" (UID: \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:47:14.580951 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.580873    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-tls-certs\") pod \"rate-test-model-c50e10-kserve-777754bcc9-wxdq8\" (UID: \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:47:14.580951 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.580930    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f65lr\" (UniqueName: \"kubernetes.io/projected/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-kube-api-access-f65lr\") pod \"rate-test-model-c50e10-kserve-777754bcc9-wxdq8\" (UID: \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:47:14.581220 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.581161    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-tmp-dir\") pod \"rate-test-model-c50e10-kserve-777754bcc9-wxdq8\" (UID: \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:47:14.581220 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.581206    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-home\") pod \"rate-test-model-c50e10-kserve-777754bcc9-wxdq8\" (UID: \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:47:14.581317 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.581246    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-model-cache\") pod \"rate-test-model-c50e10-kserve-777754bcc9-wxdq8\" (UID: \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:47:14.583180 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.583152    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-dshm\") pod \"rate-test-model-c50e10-kserve-777754bcc9-wxdq8\" (UID: \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:47:14.583412 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.583394    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-tls-certs\") pod \"rate-test-model-c50e10-kserve-777754bcc9-wxdq8\" (UID: \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:47:14.589539 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.589514    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f65lr\" (UniqueName: \"kubernetes.io/projected/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-kube-api-access-f65lr\") pod \"rate-test-model-c50e10-kserve-777754bcc9-wxdq8\" (UID: \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:47:14.694585 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.694500    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:47:14.976339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:14.976306    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"]
Sep 18 13:47:14.976804 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:47:14.976780    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82e4d85d_03b5_4dd6_aad0_86bf5300ba61.slice/crio-cf6c0751caae77f4aaf353348b3c2ed64fe07688657115b425f50f8d0599d192 WatchSource:0}: Error finding container cf6c0751caae77f4aaf353348b3c2ed64fe07688657115b425f50f8d0599d192: Status 404 returned error can't find the container with id cf6c0751caae77f4aaf353348b3c2ed64fe07688657115b425f50f8d0599d192
Sep 18 13:47:15.743275 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:15.743237    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8" event={"ID":"82e4d85d-03b5-4dd6-aad0-86bf5300ba61","Type":"ContainerStarted","Data":"b347569927b6024adeccbc57cb59ab716501c94c98855954a059d1c5cf527f1b"}
Sep 18 13:47:15.743682 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:15.743288    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8" event={"ID":"82e4d85d-03b5-4dd6-aad0-86bf5300ba61","Type":"ContainerStarted","Data":"cf6c0751caae77f4aaf353348b3c2ed64fe07688657115b425f50f8d0599d192"}
Sep 18 13:47:15.765647 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:15.765571    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8" podStartSLOduration=1.765557145 podStartE2EDuration="1.765557145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:47:15.763957166 +0000 UTC m=+1944.198833570" watchObservedRunningTime="2026-09-18 13:47:15.765557145 +0000 UTC m=+1944.200433549"
Sep 18 13:47:24.694867 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:24.694822    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:47:24.694867 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:24.694879    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:47:24.707684 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:24.707655    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:47:24.802337 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:24.802307    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:47:48.724572 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:48.724488    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"]
Sep 18 13:47:55.910993 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:55.910956    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp"]
Sep 18 13:47:55.914579 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:55.914531    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp"
Sep 18 13:47:55.916835 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:55.916806    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2dd8de\""
Sep 18 13:47:55.916835 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:55.916818    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2dd8de-dockercfg-ddj6h\""
Sep 18 13:47:55.933412 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:55.933379    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp"]
Sep 18 13:47:56.055798 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:56.055759    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c7bd6a3-a76f-4361-82f9-6e25931d48f6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp\" (UID: \"8c7bd6a3-a76f-4361-82f9-6e25931d48f6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp"
Sep 18 13:47:56.055995 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:56.055825    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8x87\" (UniqueName: \"kubernetes.io/projected/8c7bd6a3-a76f-4361-82f9-6e25931d48f6-kube-api-access-k8x87\") pod \"payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp\" (UID: \"8c7bd6a3-a76f-4361-82f9-6e25931d48f6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp"
Sep 18 13:47:56.156435 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:56.156400    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c7bd6a3-a76f-4361-82f9-6e25931d48f6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp\" (UID: \"8c7bd6a3-a76f-4361-82f9-6e25931d48f6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp"
Sep 18 13:47:56.156631 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:56.156461    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8x87\" (UniqueName: \"kubernetes.io/projected/8c7bd6a3-a76f-4361-82f9-6e25931d48f6-kube-api-access-k8x87\") pod \"payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp\" (UID: \"8c7bd6a3-a76f-4361-82f9-6e25931d48f6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp"
Sep 18 13:47:56.157172 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:56.157150    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c7bd6a3-a76f-4361-82f9-6e25931d48f6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp\" (UID: \"8c7bd6a3-a76f-4361-82f9-6e25931d48f6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp"
Sep 18 13:47:56.165216 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:56.165145    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8x87\" (UniqueName: \"kubernetes.io/projected/8c7bd6a3-a76f-4361-82f9-6e25931d48f6-kube-api-access-k8x87\") pod \"payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp\" (UID: \"8c7bd6a3-a76f-4361-82f9-6e25931d48f6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp"
Sep 18 13:47:56.231040 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:56.231000    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp"
Sep 18 13:47:56.507123 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:56.507093    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp"]
Sep 18 13:47:56.509266 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:47:56.509235    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c7bd6a3_a76f_4361_82f9_6e25931d48f6.slice/crio-0ad74e5af71b6a9457bbe588c54e3f7baef549742c7fb18a92709df97846bd21 WatchSource:0}: Error finding container 0ad74e5af71b6a9457bbe588c54e3f7baef549742c7fb18a92709df97846bd21: Status 404 returned error can't find the container with id 0ad74e5af71b6a9457bbe588c54e3f7baef549742c7fb18a92709df97846bd21
Sep 18 13:47:56.962185 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:56.962144    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp" event={"ID":"8c7bd6a3-a76f-4361-82f9-6e25931d48f6","Type":"ContainerStarted","Data":"9562c2ad6b8581f838f30bd5a12280995d9a449af82d9af377191dcc34ef0be7"}
Sep 18 13:47:56.962696 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:56.962196    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp" event={"ID":"8c7bd6a3-a76f-4361-82f9-6e25931d48f6","Type":"ContainerStarted","Data":"0ad74e5af71b6a9457bbe588c54e3f7baef549742c7fb18a92709df97846bd21"}
Sep 18 13:47:56.962696 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:56.962226    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp"
Sep 18 13:47:56.978421 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:47:56.978367    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp" podStartSLOduration=1.978345809 podStartE2EDuration="1.978345809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:47:56.977246169 +0000 UTC m=+1985.412122582" watchObservedRunningTime="2026-09-18 13:47:56.978345809 +0000 UTC m=+1985.413222215"
Sep 18 13:48:03.743286 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:03.743219    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8" podUID="82e4d85d-03b5-4dd6-aad0-86bf5300ba61" containerName="main" containerID="cri-o://b347569927b6024adeccbc57cb59ab716501c94c98855954a059d1c5cf527f1b" gracePeriod=45
Sep 18 13:48:03.998935 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:03.998863    2566 generic.go:356] "Generic (PLEG): container finished" podID="82e4d85d-03b5-4dd6-aad0-86bf5300ba61" containerID="b347569927b6024adeccbc57cb59ab716501c94c98855954a059d1c5cf527f1b" exitCode=0
Sep 18 13:48:03.999070 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:03.998938    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8" event={"ID":"82e4d85d-03b5-4dd6-aad0-86bf5300ba61","Type":"ContainerDied","Data":"b347569927b6024adeccbc57cb59ab716501c94c98855954a059d1c5cf527f1b"}
Sep 18 13:48:03.999070 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:03.998987    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8" event={"ID":"82e4d85d-03b5-4dd6-aad0-86bf5300ba61","Type":"ContainerDied","Data":"cf6c0751caae77f4aaf353348b3c2ed64fe07688657115b425f50f8d0599d192"}
Sep 18 13:48:03.999070 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:03.999003    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf6c0751caae77f4aaf353348b3c2ed64fe07688657115b425f50f8d0599d192"
Sep 18 13:48:04.001133 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:04.001116    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:48:04.018398 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:04.018376    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-model-cache\") pod \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\" (UID: \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\") "
Sep 18 13:48:04.018526 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:04.018417    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-dshm\") pod \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\" (UID: \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\") "
Sep 18 13:48:04.018526 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:04.018482    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-tmp-dir\") pod \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\" (UID: \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\") "
Sep 18 13:48:04.018686 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:04.018542    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-home\") pod \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\" (UID: \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\") "
Sep 18 13:48:04.018686 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:04.018566    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f65lr\" (UniqueName: \"kubernetes.io/projected/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-kube-api-access-f65lr\") pod \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\" (UID: \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\") "
Sep 18 13:48:04.018686 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:04.018627    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-tls-certs\") pod \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\" (UID: \"82e4d85d-03b5-4dd6-aad0-86bf5300ba61\") "
Sep 18 13:48:04.018836 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:04.018713    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-model-cache" (OuterVolumeSpecName: "model-cache") pod "82e4d85d-03b5-4dd6-aad0-86bf5300ba61" (UID: "82e4d85d-03b5-4dd6-aad0-86bf5300ba61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:48:04.018836 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:04.018752    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "82e4d85d-03b5-4dd6-aad0-86bf5300ba61" (UID: "82e4d85d-03b5-4dd6-aad0-86bf5300ba61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:48:04.018836 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:04.018820    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-home" (OuterVolumeSpecName: "home") pod "82e4d85d-03b5-4dd6-aad0-86bf5300ba61" (UID: "82e4d85d-03b5-4dd6-aad0-86bf5300ba61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:48:04.019013 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:04.018990    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-home\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:48:04.019066 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:04.019018    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-model-cache\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:48:04.019066 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:04.019034    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-tmp-dir\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:48:04.020502 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:04.020482    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-dshm" (OuterVolumeSpecName: "dshm") pod "82e4d85d-03b5-4dd6-aad0-86bf5300ba61" (UID: "82e4d85d-03b5-4dd6-aad0-86bf5300ba61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:48:04.020723 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:04.020695    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-kube-api-access-f65lr" (OuterVolumeSpecName: "kube-api-access-f65lr") pod "82e4d85d-03b5-4dd6-aad0-86bf5300ba61" (UID: "82e4d85d-03b5-4dd6-aad0-86bf5300ba61"). InnerVolumeSpecName "kube-api-access-f65lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:48:04.020837 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:04.020778    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "82e4d85d-03b5-4dd6-aad0-86bf5300ba61" (UID: "82e4d85d-03b5-4dd6-aad0-86bf5300ba61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:48:04.120093 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:04.120047    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-dshm\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:48:04.120093 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:04.120085    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f65lr\" (UniqueName: \"kubernetes.io/projected/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-kube-api-access-f65lr\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:48:04.120093 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:04.120099    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e4d85d-03b5-4dd6-aad0-86bf5300ba61-tls-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:48:05.003751 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:05.003719    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"
Sep 18 13:48:05.548080 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:05.548036    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"]
Sep 18 13:48:05.774081 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:05.774051    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c50e10/rate-test-model-c50e10-kserve-777754bcc9-wxdq8"]
Sep 18 13:48:06.077489 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:06.077457    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82e4d85d-03b5-4dd6-aad0-86bf5300ba61" path="/var/lib/kubelet/pods/82e4d85d-03b5-4dd6-aad0-86bf5300ba61/volumes"
Sep 18 13:48:07.969496 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:07.969463    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp"
Sep 18 13:48:29.212952 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.212912    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp"]
Sep 18 13:48:29.213435 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.213294    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82e4d85d-03b5-4dd6-aad0-86bf5300ba61" containerName="main"
Sep 18 13:48:29.213435 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.213304    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="82e4d85d-03b5-4dd6-aad0-86bf5300ba61" containerName="main"
Sep 18 13:48:29.213435 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.213382    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="82e4d85d-03b5-4dd6-aad0-86bf5300ba61" containerName="main"
Sep 18 13:48:29.215457 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.215440    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp"
Sep 18 13:48:29.219157 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.219135    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f629bd-dockercfg-t6xff\""
Sep 18 13:48:29.227071 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.227049    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f629bd\""
Sep 18 13:48:29.248476 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.248448    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp"]
Sep 18 13:48:29.297857 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.297817    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql"]
Sep 18 13:48:29.300707 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.300681    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql"
Sep 18 13:48:29.324765 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.324725    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql"]
Sep 18 13:48:29.345759 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.345725    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7074d2cc-142b-45a9-8ba3-69213d55eb54-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp\" (UID: \"7074d2cc-142b-45a9-8ba3-69213d55eb54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp"
Sep 18 13:48:29.345759 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.345760    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm5p9\" (UniqueName: \"kubernetes.io/projected/7074d2cc-142b-45a9-8ba3-69213d55eb54-kube-api-access-lm5p9\") pod \"payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp\" (UID: \"7074d2cc-142b-45a9-8ba3-69213d55eb54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp"
Sep 18 13:48:29.446855 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.446811    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7074d2cc-142b-45a9-8ba3-69213d55eb54-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp\" (UID: \"7074d2cc-142b-45a9-8ba3-69213d55eb54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp"
Sep 18 13:48:29.446855 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.446853    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm5p9\" (UniqueName: \"kubernetes.io/projected/7074d2cc-142b-45a9-8ba3-69213d55eb54-kube-api-access-lm5p9\") pod \"payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp\" (UID: \"7074d2cc-142b-45a9-8ba3-69213d55eb54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp"
Sep 18 13:48:29.447103 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.446897    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8eb651c-9858-45cb-98d1-7ad26c9b8a17-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql\" (UID: \"c8eb651c-9858-45cb-98d1-7ad26c9b8a17\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql"
Sep 18 13:48:29.447103 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.446924    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftvn8\" (UniqueName: \"kubernetes.io/projected/c8eb651c-9858-45cb-98d1-7ad26c9b8a17-kube-api-access-ftvn8\") pod \"payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql\" (UID: \"c8eb651c-9858-45cb-98d1-7ad26c9b8a17\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql"
Sep 18 13:48:29.447568 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.447547    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7074d2cc-142b-45a9-8ba3-69213d55eb54-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp\" (UID: \"7074d2cc-142b-45a9-8ba3-69213d55eb54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp"
Sep 18 13:48:29.455463 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.455430    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm5p9\" (UniqueName: \"kubernetes.io/projected/7074d2cc-142b-45a9-8ba3-69213d55eb54-kube-api-access-lm5p9\") pod \"payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp\" (UID: \"7074d2cc-142b-45a9-8ba3-69213d55eb54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp"
Sep 18 13:48:29.525289 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.525190    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp"
Sep 18 13:48:29.548192 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.548156    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8eb651c-9858-45cb-98d1-7ad26c9b8a17-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql\" (UID: \"c8eb651c-9858-45cb-98d1-7ad26c9b8a17\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql"
Sep 18 13:48:29.548339 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.548215    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftvn8\" (UniqueName: \"kubernetes.io/projected/c8eb651c-9858-45cb-98d1-7ad26c9b8a17-kube-api-access-ftvn8\") pod \"payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql\" (UID: \"c8eb651c-9858-45cb-98d1-7ad26c9b8a17\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql"
Sep 18 13:48:29.548994 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.548971    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8eb651c-9858-45cb-98d1-7ad26c9b8a17-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql\" (UID: \"c8eb651c-9858-45cb-98d1-7ad26c9b8a17\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql"
Sep 18 13:48:29.557702 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.557668    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftvn8\" (UniqueName: \"kubernetes.io/projected/c8eb651c-9858-45cb-98d1-7ad26c9b8a17-kube-api-access-ftvn8\") pod \"payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql\" (UID: \"c8eb651c-9858-45cb-98d1-7ad26c9b8a17\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql"
Sep 18 13:48:29.614656 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.614624    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql"
Sep 18 13:48:29.799418 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.799393    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp"]
Sep 18 13:48:29.801968 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:48:29.801942    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7074d2cc_142b_45a9_8ba3_69213d55eb54.slice/crio-ebae11f89c22b9b97bdc6777d7b8dc90b29b29984939cd1aad67cdf2a7ab817c WatchSource:0}: Error finding container ebae11f89c22b9b97bdc6777d7b8dc90b29b29984939cd1aad67cdf2a7ab817c: Status 404 returned error can't find the container with id ebae11f89c22b9b97bdc6777d7b8dc90b29b29984939cd1aad67cdf2a7ab817c
Sep 18 13:48:29.895618 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:29.895565    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql"]
Sep 18 13:48:29.897066 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:48:29.897035    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8eb651c_9858_45cb_98d1_7ad26c9b8a17.slice/crio-29afa0d795b18a1b930860c114abf347ced4b4bc044f558d603b144e6490a682 WatchSource:0}: Error finding container 29afa0d795b18a1b930860c114abf347ced4b4bc044f558d603b144e6490a682: Status 404 returned error can't find the container with id 29afa0d795b18a1b930860c114abf347ced4b4bc044f558d603b144e6490a682
Sep 18 13:48:30.129028 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:30.128980    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp" event={"ID":"7074d2cc-142b-45a9-8ba3-69213d55eb54","Type":"ContainerStarted","Data":"24e86d9f6285abac40880093843a391fd3f6a15191618c7b6a3cdd90b581f4e2"}
Sep 18 13:48:30.129234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:30.129052    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp" event={"ID":"7074d2cc-142b-45a9-8ba3-69213d55eb54","Type":"ContainerStarted","Data":"ebae11f89c22b9b97bdc6777d7b8dc90b29b29984939cd1aad67cdf2a7ab817c"}
Sep 18 13:48:30.129234 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:30.129084    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp"
Sep 18 13:48:30.130744 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:30.130718    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql" event={"ID":"c8eb651c-9858-45cb-98d1-7ad26c9b8a17","Type":"ContainerStarted","Data":"3fcc402a417341f87ed1df3ef5e0edb191d2655dbc9ae8c065f600243eeded4e"}
Sep 18 13:48:30.130857 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:30.130763    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql" event={"ID":"c8eb651c-9858-45cb-98d1-7ad26c9b8a17","Type":"ContainerStarted","Data":"29afa0d795b18a1b930860c114abf347ced4b4bc044f558d603b144e6490a682"}
Sep 18 13:48:30.130857 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:30.130783    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql"
Sep 18 13:48:30.154438 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:30.152548    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp" podStartSLOduration=1.152528666 podStartE2EDuration="1.152528666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:48:30.15018471 +0000 UTC m=+2018.585061115" watchObservedRunningTime="2026-09-18 13:48:30.152528666 +0000 UTC m=+2018.587405070"
Sep 18 13:48:30.179617 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:30.179543    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql" podStartSLOduration=1.179527899 podStartE2EDuration="1.179527899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:48:30.178135362 +0000 UTC m=+2018.613011765" watchObservedRunningTime="2026-09-18 13:48:30.179527899 +0000 UTC m=+2018.614404303"
Sep 18 13:48:41.138752 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:41.138718    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp"
Sep 18 13:48:41.139185 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:41.138823    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql"
Sep 18 13:48:52.495387 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:52.495352    2566 scope.go:117] "RemoveContainer" containerID="5a341bf05713ccb27a4fa9141b912974a66b4e09c5ac69c03fceb56ebcd4aa29"
Sep 18 13:48:58.610514 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:48:58.610483    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql_c8eb651c-9858-45cb-98d1-7ad26c9b8a17/payload-processing/0.log"
Sep 18 13:49:29.633332 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:29.633299    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp_8c7bd6a3-a76f-4361-82f9-6e25931d48f6/payload-processing/0.log"
Sep 18 13:49:29.882528 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:29.882468    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql_c8eb651c-9858-45cb-98d1-7ad26c9b8a17/payload-processing/0.log"
Sep 18 13:49:30.442097 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.442055    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"]
Sep 18 13:49:30.448840 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.448814    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.453123 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.453092    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-eaff22-openshift-default-dockercfg-gfnkq\""
Sep 18 13:49:30.466961 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.466935    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"]
Sep 18 13:49:30.505169 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.505140    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a231903-13bc-48f0-ad95-a5ac0da21b95-istio-token\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.505343 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.505195    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-istio-envoy\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.505343 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.505213    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a231903-13bc-48f0-ad95-a5ac0da21b95-istio-podinfo\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.505343 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.505241    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a231903-13bc-48f0-ad95-a5ac0da21b95-istiod-ca-cert\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.505343 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.505257    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-workload-certs\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.505343 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.505284    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d548h\" (UniqueName: \"kubernetes.io/projected/2a231903-13bc-48f0-ad95-a5ac0da21b95-kube-api-access-d548h\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.505618 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.505372    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-workload-socket\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.505618 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.505416    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-istio-data\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.505618 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.505473    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-credential-socket\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.606574 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.606533    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a231903-13bc-48f0-ad95-a5ac0da21b95-istiod-ca-cert\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.606574 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.606573    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-workload-certs\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.606864 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.606640    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d548h\" (UniqueName: \"kubernetes.io/projected/2a231903-13bc-48f0-ad95-a5ac0da21b95-kube-api-access-d548h\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.606864 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.606676    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-workload-socket\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.606864 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.606822    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-istio-data\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.606864 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.606861    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-credential-socket\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.607088 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.606901    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a231903-13bc-48f0-ad95-a5ac0da21b95-istio-token\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.607088 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.606975    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-istio-envoy\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.607088 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.606997    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a231903-13bc-48f0-ad95-a5ac0da21b95-istio-podinfo\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.607088 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.607068    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-workload-certs\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.607307 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.607112    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-workload-socket\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.607307 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.607167    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-istio-data\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.607419 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.607339    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a231903-13bc-48f0-ad95-a5ac0da21b95-istiod-ca-cert\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.607419 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.607381    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-credential-socket\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.609318 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.609293    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-istio-envoy\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.609527 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.609511    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a231903-13bc-48f0-ad95-a5ac0da21b95-istio-podinfo\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.625015 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.624993    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a231903-13bc-48f0-ad95-a5ac0da21b95-istio-token\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.625356 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.625336    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d548h\" (UniqueName: \"kubernetes.io/projected/2a231903-13bc-48f0-ad95-a5ac0da21b95-kube-api-access-d548h\") pod \"e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") " pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:30.759070 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:30.758979    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:31.058515 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:31.058486    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"]
Sep 18 13:49:31.059761 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:49:31.059729    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a231903_13bc_48f0_ad95_a5ac0da21b95.slice/crio-23ce7845de5410192a382ebb64953d637318c83306ea7f726820daa80422646b WatchSource:0}: Error finding container 23ce7845de5410192a382ebb64953d637318c83306ea7f726820daa80422646b: Status 404 returned error can't find the container with id 23ce7845de5410192a382ebb64953d637318c83306ea7f726820daa80422646b
Sep 18 13:49:31.062064 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:31.062030    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:49:31.062178 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:31.062113    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:49:31.062178 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:31.062160    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:49:31.451480 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:31.451448    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9" event={"ID":"2a231903-13bc-48f0-ad95-a5ac0da21b95","Type":"ContainerStarted","Data":"69a2b5a678703608b2d60714d4f60b36c486a3bf73e4ec83095f8c722ac18dd2"}
Sep 18 13:49:31.451687 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:31.451505    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9" event={"ID":"2a231903-13bc-48f0-ad95-a5ac0da21b95","Type":"ContainerStarted","Data":"23ce7845de5410192a382ebb64953d637318c83306ea7f726820daa80422646b"}
Sep 18 13:49:31.479866 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:31.479814    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9" podStartSLOduration=1.479798543 podStartE2EDuration="1.479798543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:49:31.477820864 +0000 UTC m=+2079.912697267" watchObservedRunningTime="2026-09-18 13:49:31.479798543 +0000 UTC m=+2079.914674947"
Sep 18 13:49:31.759719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:31.759616    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:32.765647 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:32.765613    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:33.461466 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:33.461438    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:33.462370 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:33.462350    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:52.172798 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:52.172757    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k86vn_55e33e14-3f7a-4793-a724-9fd5022740d7/console-operator/1.log"
Sep 18 13:49:52.176742 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:52.176710    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k86vn_55e33e14-3f7a-4793-a724-9fd5022740d7/console-operator/1.log"
Sep 18 13:49:52.516168 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:52.516074    2566 scope.go:117] "RemoveContainer" containerID="48278f777fb9c76e47a9c9902d56ae775b5ff727a4cb6f5c24e58aa4f457415d"
Sep 18 13:49:53.566127 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:53.566088    2566 generic.go:356] "Generic (PLEG): container finished" podID="2a231903-13bc-48f0-ad95-a5ac0da21b95" containerID="69a2b5a678703608b2d60714d4f60b36c486a3bf73e4ec83095f8c722ac18dd2" exitCode=0
Sep 18 13:49:53.566564 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:53.566165    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9" event={"ID":"2a231903-13bc-48f0-ad95-a5ac0da21b95","Type":"ContainerDied","Data":"69a2b5a678703608b2d60714d4f60b36c486a3bf73e4ec83095f8c722ac18dd2"}
Sep 18 13:49:53.566660 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:53.566645    2566 scope.go:117] "RemoveContainer" containerID="69a2b5a678703608b2d60714d4f60b36c486a3bf73e4ec83095f8c722ac18dd2"
Sep 18 13:49:53.567806 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:53.567771    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:49:53.567875 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:53.567860    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:49:53.567919 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:53.567898    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 13:49:54.573034 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:54.573002    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9" event={"ID":"2a231903-13bc-48f0-ad95-a5ac0da21b95","Type":"ContainerStarted","Data":"5f85f5c324562750692495e62e549838a7f9c339d176581b9c92b1a30df29e9d"}
Sep 18 13:49:54.759696 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:54.759657    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:49:54.760094 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:54.759995    2566 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Sep 18 13:49:54.760271 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:54.760248    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9" podUID="2a231903-13bc-48f0-ad95-a5ac0da21b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 18 13:49:55.760140 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:55.760091    2566 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Sep 18 13:49:55.760727 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:55.760161    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9" podUID="2a231903-13bc-48f0-ad95-a5ac0da21b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 18 13:49:56.760146 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:56.760099    2566 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Sep 18 13:49:56.760472 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:56.760437    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9" podUID="2a231903-13bc-48f0-ad95-a5ac0da21b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 18 13:49:57.759791 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:57.759744    2566 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Sep 18 13:49:57.760020 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:57.759821    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9" podUID="2a231903-13bc-48f0-ad95-a5ac0da21b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 18 13:49:58.760396 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:58.760354    2566 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Sep 18 13:49:58.761061 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:58.761024    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9" podUID="2a231903-13bc-48f0-ad95-a5ac0da21b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 18 13:49:59.759836 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:59.759797    2566 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Sep 18 13:49:59.760176 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:49:59.760145    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9" podUID="2a231903-13bc-48f0-ad95-a5ac0da21b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 18 13:50:00.764492 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:00.764461    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:50:00.764997 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:00.764799    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:50:00.765344 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:00.765324    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:50:04.353680 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:04.353637    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eaff22-p2xfn"]
Sep 18 13:50:04.356539 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:04.356518    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eaff22-p2xfn"
Sep 18 13:50:04.358742 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:04.358717    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nh2cd\""
Sep 18 13:50:04.358866 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:04.358717    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:50:04.375356 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:04.375330    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eaff22-p2xfn"]
Sep 18 13:50:04.549779 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:04.549743    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c7ef8d-d9a3-4ae7-b7c3-aa66860fa6cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-eaff22-p2xfn\" (UID: \"95c7ef8d-d9a3-4ae7-b7c3-aa66860fa6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eaff22-p2xfn"
Sep 18 13:50:04.650695 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:04.650580    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c7ef8d-d9a3-4ae7-b7c3-aa66860fa6cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-eaff22-p2xfn\" (UID: \"95c7ef8d-d9a3-4ae7-b7c3-aa66860fa6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eaff22-p2xfn"
Sep 18 13:50:04.651231 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:04.651212    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c7ef8d-d9a3-4ae7-b7c3-aa66860fa6cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-eaff22-p2xfn\" (UID: \"95c7ef8d-d9a3-4ae7-b7c3-aa66860fa6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eaff22-p2xfn"
Sep 18 13:50:04.666655 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:04.666618    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eaff22-p2xfn"
Sep 18 13:50:04.952610 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:04.952554    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eaff22-p2xfn"]
Sep 18 13:50:04.954193 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:50:04.954165    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c7ef8d_d9a3_4ae7_b7c3_aa66860fa6cb.slice/crio-43ee5736898d12dd1f0f0fa6aa84aa032dd619c3c07cab19916ced86d74dcc14 WatchSource:0}: Error finding container 43ee5736898d12dd1f0f0fa6aa84aa032dd619c3c07cab19916ced86d74dcc14: Status 404 returned error can't find the container with id 43ee5736898d12dd1f0f0fa6aa84aa032dd619c3c07cab19916ced86d74dcc14
Sep 18 13:50:05.642398 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:05.642364    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eaff22-p2xfn" event={"ID":"95c7ef8d-d9a3-4ae7-b7c3-aa66860fa6cb","Type":"ContainerStarted","Data":"b75e49238152b3c9e50b59a5c6a14a7d4ae9d2fae9e916c192597b5aba660152"}
Sep 18 13:50:05.642807 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:05.642419    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eaff22-p2xfn" event={"ID":"95c7ef8d-d9a3-4ae7-b7c3-aa66860fa6cb","Type":"ContainerStarted","Data":"43ee5736898d12dd1f0f0fa6aa84aa032dd619c3c07cab19916ced86d74dcc14"}
Sep 18 13:50:05.667552 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:05.667502    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eaff22-p2xfn" podStartSLOduration=1.667486479 podStartE2EDuration="1.667486479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:50:05.666630039 +0000 UTC m=+2114.101506444" watchObservedRunningTime="2026-09-18 13:50:05.667486479 +0000 UTC m=+2114.102362941"
Sep 18 13:50:06.649291 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:06.649255    2566 generic.go:356] "Generic (PLEG): container finished" podID="95c7ef8d-d9a3-4ae7-b7c3-aa66860fa6cb" containerID="b75e49238152b3c9e50b59a5c6a14a7d4ae9d2fae9e916c192597b5aba660152" exitCode=0
Sep 18 13:50:06.649716 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:06.649349    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eaff22-p2xfn" event={"ID":"95c7ef8d-d9a3-4ae7-b7c3-aa66860fa6cb","Type":"ContainerDied","Data":"b75e49238152b3c9e50b59a5c6a14a7d4ae9d2fae9e916c192597b5aba660152"}
Sep 18 13:50:07.791634 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:07.791607    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eaff22-p2xfn"
Sep 18 13:50:07.879914 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:07.879873    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c7ef8d-d9a3-4ae7-b7c3-aa66860fa6cb-maas-api-service-ca\") pod \"95c7ef8d-d9a3-4ae7-b7c3-aa66860fa6cb\" (UID: \"95c7ef8d-d9a3-4ae7-b7c3-aa66860fa6cb\") "
Sep 18 13:50:07.880228 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:07.880204    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c7ef8d-d9a3-4ae7-b7c3-aa66860fa6cb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "95c7ef8d-d9a3-4ae7-b7c3-aa66860fa6cb" (UID: "95c7ef8d-d9a3-4ae7-b7c3-aa66860fa6cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:50:07.980980 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:07.980900    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c7ef8d-d9a3-4ae7-b7c3-aa66860fa6cb-maas-api-service-ca\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:50:08.661637 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:08.661586    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eaff22-p2xfn"
Sep 18 13:50:08.661817 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:08.661635    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eaff22-p2xfn" event={"ID":"95c7ef8d-d9a3-4ae7-b7c3-aa66860fa6cb","Type":"ContainerDied","Data":"43ee5736898d12dd1f0f0fa6aa84aa032dd619c3c07cab19916ced86d74dcc14"}
Sep 18 13:50:08.661817 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:08.661691    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43ee5736898d12dd1f0f0fa6aa84aa032dd619c3c07cab19916ced86d74dcc14"
Sep 18 13:50:16.309512 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:16.309474    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eaff22-p2xfn"]
Sep 18 13:50:16.312976 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:16.312938    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eaff22-p2xfn"]
Sep 18 13:50:18.077509 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:18.077470    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95c7ef8d-d9a3-4ae7-b7c3-aa66860fa6cb" path="/var/lib/kubelet/pods/95c7ef8d-d9a3-4ae7-b7c3-aa66860fa6cb/volumes"
Sep 18 13:50:23.823422 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:23.823385    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"]
Sep 18 13:50:23.823934 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:23.823626    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9" podUID="2a231903-13bc-48f0-ad95-a5ac0da21b95" containerName="istio-proxy" containerID="cri-o://5f85f5c324562750692495e62e549838a7f9c339d176581b9c92b1a30df29e9d" gracePeriod=30
Sep 18 13:50:29.080639 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.080587    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:50:29.183144 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.183113    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-istio-data\") pod \"2a231903-13bc-48f0-ad95-a5ac0da21b95\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") "
Sep 18 13:50:29.183334 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.183167    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-istio-envoy\") pod \"2a231903-13bc-48f0-ad95-a5ac0da21b95\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") "
Sep 18 13:50:29.183334 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.183208    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a231903-13bc-48f0-ad95-a5ac0da21b95-istio-token\") pod \"2a231903-13bc-48f0-ad95-a5ac0da21b95\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") "
Sep 18 13:50:29.183334 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.183227    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a231903-13bc-48f0-ad95-a5ac0da21b95-istio-podinfo\") pod \"2a231903-13bc-48f0-ad95-a5ac0da21b95\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") "
Sep 18 13:50:29.183334 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.183263    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a231903-13bc-48f0-ad95-a5ac0da21b95-istiod-ca-cert\") pod \"2a231903-13bc-48f0-ad95-a5ac0da21b95\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") "
Sep 18 13:50:29.183334 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.183311    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-workload-socket\") pod \"2a231903-13bc-48f0-ad95-a5ac0da21b95\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") "
Sep 18 13:50:29.183623 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.183346    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d548h\" (UniqueName: \"kubernetes.io/projected/2a231903-13bc-48f0-ad95-a5ac0da21b95-kube-api-access-d548h\") pod \"2a231903-13bc-48f0-ad95-a5ac0da21b95\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") "
Sep 18 13:50:29.183623 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.183395    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-workload-certs\") pod \"2a231903-13bc-48f0-ad95-a5ac0da21b95\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") "
Sep 18 13:50:29.183623 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.183430    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-credential-socket\") pod \"2a231903-13bc-48f0-ad95-a5ac0da21b95\" (UID: \"2a231903-13bc-48f0-ad95-a5ac0da21b95\") "
Sep 18 13:50:29.183623 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.183434    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-istio-data" (OuterVolumeSpecName: "istio-data") pod "2a231903-13bc-48f0-ad95-a5ac0da21b95" (UID: "2a231903-13bc-48f0-ad95-a5ac0da21b95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:50:29.183839 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.183700    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a231903-13bc-48f0-ad95-a5ac0da21b95-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2a231903-13bc-48f0-ad95-a5ac0da21b95" (UID: "2a231903-13bc-48f0-ad95-a5ac0da21b95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:50:29.183839 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.183790    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-istio-data\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:50:29.183839 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.183809    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a231903-13bc-48f0-ad95-a5ac0da21b95-istiod-ca-cert\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:50:29.183992 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.183924    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2a231903-13bc-48f0-ad95-a5ac0da21b95" (UID: "2a231903-13bc-48f0-ad95-a5ac0da21b95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:50:29.184056 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.184020    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2a231903-13bc-48f0-ad95-a5ac0da21b95" (UID: "2a231903-13bc-48f0-ad95-a5ac0da21b95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:50:29.184103 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.183849    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2a231903-13bc-48f0-ad95-a5ac0da21b95" (UID: "2a231903-13bc-48f0-ad95-a5ac0da21b95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:50:29.185914 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.185889    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a231903-13bc-48f0-ad95-a5ac0da21b95-istio-token" (OuterVolumeSpecName: "istio-token") pod "2a231903-13bc-48f0-ad95-a5ac0da21b95" (UID: "2a231903-13bc-48f0-ad95-a5ac0da21b95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:50:29.185914 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.185889    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2a231903-13bc-48f0-ad95-a5ac0da21b95" (UID: "2a231903-13bc-48f0-ad95-a5ac0da21b95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:50:29.185914 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.185918    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a231903-13bc-48f0-ad95-a5ac0da21b95-kube-api-access-d548h" (OuterVolumeSpecName: "kube-api-access-d548h") pod "2a231903-13bc-48f0-ad95-a5ac0da21b95" (UID: "2a231903-13bc-48f0-ad95-a5ac0da21b95"). InnerVolumeSpecName "kube-api-access-d548h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:50:29.186184 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.186129    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a231903-13bc-48f0-ad95-a5ac0da21b95-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2a231903-13bc-48f0-ad95-a5ac0da21b95" (UID: "2a231903-13bc-48f0-ad95-a5ac0da21b95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 13:50:29.285101 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.285045    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-workload-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:50:29.285101 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.285096    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d548h\" (UniqueName: \"kubernetes.io/projected/2a231903-13bc-48f0-ad95-a5ac0da21b95-kube-api-access-d548h\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:50:29.285101 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.285106    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-workload-certs\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:50:29.285101 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.285114    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-credential-socket\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:50:29.285101 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.285124    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a231903-13bc-48f0-ad95-a5ac0da21b95-istio-envoy\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:50:29.285439 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.285132    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a231903-13bc-48f0-ad95-a5ac0da21b95-istio-token\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:50:29.285439 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.285140    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a231903-13bc-48f0-ad95-a5ac0da21b95-istio-podinfo\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:50:29.775013 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.774975    2566 generic.go:356] "Generic (PLEG): container finished" podID="2a231903-13bc-48f0-ad95-a5ac0da21b95" containerID="5f85f5c324562750692495e62e549838a7f9c339d176581b9c92b1a30df29e9d" exitCode=0
Sep 18 13:50:29.775206 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.775035    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"
Sep 18 13:50:29.775206 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.775036    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9" event={"ID":"2a231903-13bc-48f0-ad95-a5ac0da21b95","Type":"ContainerDied","Data":"5f85f5c324562750692495e62e549838a7f9c339d176581b9c92b1a30df29e9d"}
Sep 18 13:50:29.775206 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.775155    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9" event={"ID":"2a231903-13bc-48f0-ad95-a5ac0da21b95","Type":"ContainerDied","Data":"23ce7845de5410192a382ebb64953d637318c83306ea7f726820daa80422646b"}
Sep 18 13:50:29.775206 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.775177    2566 scope.go:117] "RemoveContainer" containerID="5f85f5c324562750692495e62e549838a7f9c339d176581b9c92b1a30df29e9d"
Sep 18 13:50:29.784582 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.784565    2566 scope.go:117] "RemoveContainer" containerID="69a2b5a678703608b2d60714d4f60b36c486a3bf73e4ec83095f8c722ac18dd2"
Sep 18 13:50:29.793258 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.793237    2566 scope.go:117] "RemoveContainer" containerID="5f85f5c324562750692495e62e549838a7f9c339d176581b9c92b1a30df29e9d"
Sep 18 13:50:29.793531 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:50:29.793510    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f85f5c324562750692495e62e549838a7f9c339d176581b9c92b1a30df29e9d\": container with ID starting with 5f85f5c324562750692495e62e549838a7f9c339d176581b9c92b1a30df29e9d not found: ID does not exist" containerID="5f85f5c324562750692495e62e549838a7f9c339d176581b9c92b1a30df29e9d"
Sep 18 13:50:29.793648 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.793542    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f85f5c324562750692495e62e549838a7f9c339d176581b9c92b1a30df29e9d"} err="failed to get container status \"5f85f5c324562750692495e62e549838a7f9c339d176581b9c92b1a30df29e9d\": rpc error: code = NotFound desc = could not find container \"5f85f5c324562750692495e62e549838a7f9c339d176581b9c92b1a30df29e9d\": container with ID starting with 5f85f5c324562750692495e62e549838a7f9c339d176581b9c92b1a30df29e9d not found: ID does not exist"
Sep 18 13:50:29.793648 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.793567    2566 scope.go:117] "RemoveContainer" containerID="69a2b5a678703608b2d60714d4f60b36c486a3bf73e4ec83095f8c722ac18dd2"
Sep 18 13:50:29.793892 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:50:29.793875    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a2b5a678703608b2d60714d4f60b36c486a3bf73e4ec83095f8c722ac18dd2\": container with ID starting with 69a2b5a678703608b2d60714d4f60b36c486a3bf73e4ec83095f8c722ac18dd2 not found: ID does not exist" containerID="69a2b5a678703608b2d60714d4f60b36c486a3bf73e4ec83095f8c722ac18dd2"
Sep 18 13:50:29.793937 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.793899    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a2b5a678703608b2d60714d4f60b36c486a3bf73e4ec83095f8c722ac18dd2"} err="failed to get container status \"69a2b5a678703608b2d60714d4f60b36c486a3bf73e4ec83095f8c722ac18dd2\": rpc error: code = NotFound desc = could not find container \"69a2b5a678703608b2d60714d4f60b36c486a3bf73e4ec83095f8c722ac18dd2\": container with ID starting with 69a2b5a678703608b2d60714d4f60b36c486a3bf73e4ec83095f8c722ac18dd2 not found: ID does not exist"
Sep 18 13:50:29.798381 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.798359    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"]
Sep 18 13:50:29.804348 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:29.804328    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-eaff22-openshift-default-84dcb59944-lnjb9"]
Sep 18 13:50:30.076725 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:30.076620    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a231903-13bc-48f0-ad95-a5ac0da21b95" path="/var/lib/kubelet/pods/2a231903-13bc-48f0-ad95-a5ac0da21b95/volumes"
Sep 18 13:50:43.401447 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:43.401397    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp"]
Sep 18 13:50:43.401933 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:43.401634    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp" podUID="8c7bd6a3-a76f-4361-82f9-6e25931d48f6" containerName="payload-processing" containerID="cri-o://9562c2ad6b8581f838f30bd5a12280995d9a449af82d9af377191dcc34ef0be7" gracePeriod=30
Sep 18 13:50:43.662684 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:43.662608    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp"
Sep 18 13:50:43.812509 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:43.812467    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8x87\" (UniqueName: \"kubernetes.io/projected/8c7bd6a3-a76f-4361-82f9-6e25931d48f6-kube-api-access-k8x87\") pod \"8c7bd6a3-a76f-4361-82f9-6e25931d48f6\" (UID: \"8c7bd6a3-a76f-4361-82f9-6e25931d48f6\") "
Sep 18 13:50:43.812750 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:43.812552    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c7bd6a3-a76f-4361-82f9-6e25931d48f6-plugins-config-volume\") pod \"8c7bd6a3-a76f-4361-82f9-6e25931d48f6\" (UID: \"8c7bd6a3-a76f-4361-82f9-6e25931d48f6\") "
Sep 18 13:50:43.812936 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:43.812907    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c7bd6a3-a76f-4361-82f9-6e25931d48f6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8c7bd6a3-a76f-4361-82f9-6e25931d48f6" (UID: "8c7bd6a3-a76f-4361-82f9-6e25931d48f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:50:43.814675 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:43.814647    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c7bd6a3-a76f-4361-82f9-6e25931d48f6-kube-api-access-k8x87" (OuterVolumeSpecName: "kube-api-access-k8x87") pod "8c7bd6a3-a76f-4361-82f9-6e25931d48f6" (UID: "8c7bd6a3-a76f-4361-82f9-6e25931d48f6"). InnerVolumeSpecName "kube-api-access-k8x87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:50:43.847809 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:43.847768    2566 generic.go:356] "Generic (PLEG): container finished" podID="8c7bd6a3-a76f-4361-82f9-6e25931d48f6" containerID="9562c2ad6b8581f838f30bd5a12280995d9a449af82d9af377191dcc34ef0be7" exitCode=0
Sep 18 13:50:43.847976 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:43.847813    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp"
Sep 18 13:50:43.847976 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:43.847851    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp" event={"ID":"8c7bd6a3-a76f-4361-82f9-6e25931d48f6","Type":"ContainerDied","Data":"9562c2ad6b8581f838f30bd5a12280995d9a449af82d9af377191dcc34ef0be7"}
Sep 18 13:50:43.847976 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:43.847901    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp" event={"ID":"8c7bd6a3-a76f-4361-82f9-6e25931d48f6","Type":"ContainerDied","Data":"0ad74e5af71b6a9457bbe588c54e3f7baef549742c7fb18a92709df97846bd21"}
Sep 18 13:50:43.847976 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:43.847925    2566 scope.go:117] "RemoveContainer" containerID="9562c2ad6b8581f838f30bd5a12280995d9a449af82d9af377191dcc34ef0be7"
Sep 18 13:50:43.857092 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:43.857073    2566 scope.go:117] "RemoveContainer" containerID="9562c2ad6b8581f838f30bd5a12280995d9a449af82d9af377191dcc34ef0be7"
Sep 18 13:50:43.857374 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:50:43.857357    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9562c2ad6b8581f838f30bd5a12280995d9a449af82d9af377191dcc34ef0be7\": container with ID starting with 9562c2ad6b8581f838f30bd5a12280995d9a449af82d9af377191dcc34ef0be7 not found: ID does not exist" containerID="9562c2ad6b8581f838f30bd5a12280995d9a449af82d9af377191dcc34ef0be7"
Sep 18 13:50:43.857414 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:43.857382    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9562c2ad6b8581f838f30bd5a12280995d9a449af82d9af377191dcc34ef0be7"} err="failed to get container status \"9562c2ad6b8581f838f30bd5a12280995d9a449af82d9af377191dcc34ef0be7\": rpc error: code = NotFound desc = could not find container \"9562c2ad6b8581f838f30bd5a12280995d9a449af82d9af377191dcc34ef0be7\": container with ID starting with 9562c2ad6b8581f838f30bd5a12280995d9a449af82d9af377191dcc34ef0be7 not found: ID does not exist"
Sep 18 13:50:43.880924 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:43.880889    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp"]
Sep 18 13:50:43.886359 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:43.886330    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2dd8de-648db5f59f-kf7zp"]
Sep 18 13:50:43.913877 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:43.913790    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c7bd6a3-a76f-4361-82f9-6e25931d48f6-plugins-config-volume\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:50:43.913877 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:43.913822    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8x87\" (UniqueName: \"kubernetes.io/projected/8c7bd6a3-a76f-4361-82f9-6e25931d48f6-kube-api-access-k8x87\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:50:44.077397 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:50:44.077364    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c7bd6a3-a76f-4361-82f9-6e25931d48f6" path="/var/lib/kubelet/pods/8c7bd6a3-a76f-4361-82f9-6e25931d48f6/volumes"
Sep 18 13:51:34.978977 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:34.978842    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql"]
Sep 18 13:51:34.979507 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:34.979112    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql" podUID="c8eb651c-9858-45cb-98d1-7ad26c9b8a17" containerName="payload-processing" containerID="cri-o://3fcc402a417341f87ed1df3ef5e0edb191d2655dbc9ae8c065f600243eeded4e" gracePeriod=30
Sep 18 13:51:34.985862 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:34.985835    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp"]
Sep 18 13:51:34.986179 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:34.986139    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp" podUID="7074d2cc-142b-45a9-8ba3-69213d55eb54" containerName="payload-pre-processing" containerID="cri-o://24e86d9f6285abac40880093843a391fd3f6a15191618c7b6a3cdd90b581f4e2" gracePeriod=30
Sep 18 13:51:35.109920 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:35.109881    2566 generic.go:356] "Generic (PLEG): container finished" podID="c8eb651c-9858-45cb-98d1-7ad26c9b8a17" containerID="3fcc402a417341f87ed1df3ef5e0edb191d2655dbc9ae8c065f600243eeded4e" exitCode=0
Sep 18 13:51:35.110096 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:35.109950    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql" event={"ID":"c8eb651c-9858-45cb-98d1-7ad26c9b8a17","Type":"ContainerDied","Data":"3fcc402a417341f87ed1df3ef5e0edb191d2655dbc9ae8c065f600243eeded4e"}
Sep 18 13:51:35.111936 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:35.111911    2566 generic.go:356] "Generic (PLEG): container finished" podID="7074d2cc-142b-45a9-8ba3-69213d55eb54" containerID="24e86d9f6285abac40880093843a391fd3f6a15191618c7b6a3cdd90b581f4e2" exitCode=0
Sep 18 13:51:35.112063 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:35.111959    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp" event={"ID":"7074d2cc-142b-45a9-8ba3-69213d55eb54","Type":"ContainerDied","Data":"24e86d9f6285abac40880093843a391fd3f6a15191618c7b6a3cdd90b581f4e2"}
Sep 18 13:51:35.261383 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:35.261357    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql"
Sep 18 13:51:35.264561 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:35.264540    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp"
Sep 18 13:51:35.292239 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:35.292211    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm5p9\" (UniqueName: \"kubernetes.io/projected/7074d2cc-142b-45a9-8ba3-69213d55eb54-kube-api-access-lm5p9\") pod \"7074d2cc-142b-45a9-8ba3-69213d55eb54\" (UID: \"7074d2cc-142b-45a9-8ba3-69213d55eb54\") "
Sep 18 13:51:35.292395 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:35.292304    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7074d2cc-142b-45a9-8ba3-69213d55eb54-plugins-config-volume\") pod \"7074d2cc-142b-45a9-8ba3-69213d55eb54\" (UID: \"7074d2cc-142b-45a9-8ba3-69213d55eb54\") "
Sep 18 13:51:35.292395 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:35.292359    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8eb651c-9858-45cb-98d1-7ad26c9b8a17-plugins-config-volume\") pod \"c8eb651c-9858-45cb-98d1-7ad26c9b8a17\" (UID: \"c8eb651c-9858-45cb-98d1-7ad26c9b8a17\") "
Sep 18 13:51:35.292707 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:35.292682    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7074d2cc-142b-45a9-8ba3-69213d55eb54-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7074d2cc-142b-45a9-8ba3-69213d55eb54" (UID: "7074d2cc-142b-45a9-8ba3-69213d55eb54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:51:35.292789 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:35.292733    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8eb651c-9858-45cb-98d1-7ad26c9b8a17-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c8eb651c-9858-45cb-98d1-7ad26c9b8a17" (UID: "c8eb651c-9858-45cb-98d1-7ad26c9b8a17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:51:35.294422 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:35.294404    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7074d2cc-142b-45a9-8ba3-69213d55eb54-kube-api-access-lm5p9" (OuterVolumeSpecName: "kube-api-access-lm5p9") pod "7074d2cc-142b-45a9-8ba3-69213d55eb54" (UID: "7074d2cc-142b-45a9-8ba3-69213d55eb54"). InnerVolumeSpecName "kube-api-access-lm5p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:51:35.393026 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:35.392986    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftvn8\" (UniqueName: \"kubernetes.io/projected/c8eb651c-9858-45cb-98d1-7ad26c9b8a17-kube-api-access-ftvn8\") pod \"c8eb651c-9858-45cb-98d1-7ad26c9b8a17\" (UID: \"c8eb651c-9858-45cb-98d1-7ad26c9b8a17\") "
Sep 18 13:51:35.393199 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:35.393153    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8eb651c-9858-45cb-98d1-7ad26c9b8a17-plugins-config-volume\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:51:35.393199 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:35.393165    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm5p9\" (UniqueName: \"kubernetes.io/projected/7074d2cc-142b-45a9-8ba3-69213d55eb54-kube-api-access-lm5p9\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:51:35.393199 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:35.393174    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7074d2cc-142b-45a9-8ba3-69213d55eb54-plugins-config-volume\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:51:35.395104 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:35.395079    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8eb651c-9858-45cb-98d1-7ad26c9b8a17-kube-api-access-ftvn8" (OuterVolumeSpecName: "kube-api-access-ftvn8") pod "c8eb651c-9858-45cb-98d1-7ad26c9b8a17" (UID: "c8eb651c-9858-45cb-98d1-7ad26c9b8a17"). InnerVolumeSpecName "kube-api-access-ftvn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:51:35.493817 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:35.493729    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftvn8\" (UniqueName: \"kubernetes.io/projected/c8eb651c-9858-45cb-98d1-7ad26c9b8a17-kube-api-access-ftvn8\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:51:36.119048 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:36.119015    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql"
Sep 18 13:51:36.119438 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:36.119051    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql" event={"ID":"c8eb651c-9858-45cb-98d1-7ad26c9b8a17","Type":"ContainerDied","Data":"29afa0d795b18a1b930860c114abf347ced4b4bc044f558d603b144e6490a682"}
Sep 18 13:51:36.119438 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:36.119121    2566 scope.go:117] "RemoveContainer" containerID="3fcc402a417341f87ed1df3ef5e0edb191d2655dbc9ae8c065f600243eeded4e"
Sep 18 13:51:36.120517 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:36.120495    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp"
Sep 18 13:51:36.120614 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:36.120497    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp" event={"ID":"7074d2cc-142b-45a9-8ba3-69213d55eb54","Type":"ContainerDied","Data":"ebae11f89c22b9b97bdc6777d7b8dc90b29b29984939cd1aad67cdf2a7ab817c"}
Sep 18 13:51:36.127929 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:36.127911    2566 scope.go:117] "RemoveContainer" containerID="24e86d9f6285abac40880093843a391fd3f6a15191618c7b6a3cdd90b581f4e2"
Sep 18 13:51:36.149450 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:36.149428    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp"]
Sep 18 13:51:36.158570 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:36.158549    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f629bd-5f87cb6c86-wx9gp"]
Sep 18 13:51:36.188651 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:36.188625    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql"]
Sep 18 13:51:36.200215 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:36.200192    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f629bd-5559f687cc-9zkql"]
Sep 18 13:51:38.076884 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:38.076841    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7074d2cc-142b-45a9-8ba3-69213d55eb54" path="/var/lib/kubelet/pods/7074d2cc-142b-45a9-8ba3-69213d55eb54/volumes"
Sep 18 13:51:38.077407 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:38.077385    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8eb651c-9858-45cb-98d1-7ad26c9b8a17" path="/var/lib/kubelet/pods/c8eb651c-9858-45cb-98d1-7ad26c9b8a17/volumes"
Sep 18 13:51:43.140295 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.140259    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-537abe-w7849"]
Sep 18 13:51:43.140717 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.140681    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95c7ef8d-d9a3-4ae7-b7c3-aa66860fa6cb" containerName="revoke-keys"
Sep 18 13:51:43.140717 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.140694    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="95c7ef8d-d9a3-4ae7-b7c3-aa66860fa6cb" containerName="revoke-keys"
Sep 18 13:51:43.140717 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.140711    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a231903-13bc-48f0-ad95-a5ac0da21b95" containerName="istio-proxy"
Sep 18 13:51:43.140824 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.140719    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a231903-13bc-48f0-ad95-a5ac0da21b95" containerName="istio-proxy"
Sep 18 13:51:43.140824 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.140732    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8eb651c-9858-45cb-98d1-7ad26c9b8a17" containerName="payload-processing"
Sep 18 13:51:43.140824 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.140739    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8eb651c-9858-45cb-98d1-7ad26c9b8a17" containerName="payload-processing"
Sep 18 13:51:43.140824 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.140747    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c7bd6a3-a76f-4361-82f9-6e25931d48f6" containerName="payload-processing"
Sep 18 13:51:43.140824 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.140753    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c7bd6a3-a76f-4361-82f9-6e25931d48f6" containerName="payload-processing"
Sep 18 13:51:43.140824 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.140759    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7074d2cc-142b-45a9-8ba3-69213d55eb54" containerName="payload-pre-processing"
Sep 18 13:51:43.140824 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.140764    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7074d2cc-142b-45a9-8ba3-69213d55eb54" containerName="payload-pre-processing"
Sep 18 13:51:43.141046 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.140829    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a231903-13bc-48f0-ad95-a5ac0da21b95" containerName="istio-proxy"
Sep 18 13:51:43.141046 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.140836    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="95c7ef8d-d9a3-4ae7-b7c3-aa66860fa6cb" containerName="revoke-keys"
Sep 18 13:51:43.141046 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.140844    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c7bd6a3-a76f-4361-82f9-6e25931d48f6" containerName="payload-processing"
Sep 18 13:51:43.141046 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.140853    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7074d2cc-142b-45a9-8ba3-69213d55eb54" containerName="payload-pre-processing"
Sep 18 13:51:43.141046 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.140860    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8eb651c-9858-45cb-98d1-7ad26c9b8a17" containerName="payload-processing"
Sep 18 13:51:43.145522 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.145503    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-537abe-w7849"
Sep 18 13:51:43.147946 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.147923    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:51:43.148157 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.148140    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nh2cd\""
Sep 18 13:51:43.154968 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.154938    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-537abe-w7849"]
Sep 18 13:51:43.260621 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.260569    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c81840-eb6e-4f44-8b0c-27cbcacdd8d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-537abe-w7849\" (UID: \"74c81840-eb6e-4f44-8b0c-27cbcacdd8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-537abe-w7849"
Sep 18 13:51:43.361744 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.361702    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c81840-eb6e-4f44-8b0c-27cbcacdd8d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-537abe-w7849\" (UID: \"74c81840-eb6e-4f44-8b0c-27cbcacdd8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-537abe-w7849"
Sep 18 13:51:43.362367 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.362343    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c81840-eb6e-4f44-8b0c-27cbcacdd8d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-537abe-w7849\" (UID: \"74c81840-eb6e-4f44-8b0c-27cbcacdd8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-537abe-w7849"
Sep 18 13:51:43.457545 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.457434    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-537abe-w7849"
Sep 18 13:51:43.730276 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:43.730247    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-537abe-w7849"]
Sep 18 13:51:43.732576 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:51:43.732544    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c81840_eb6e_4f44_8b0c_27cbcacdd8d3.slice/crio-f4af87be24685fe1e9bee397fc883ce7d13f410e8c19c6c77f167426650fb476 WatchSource:0}: Error finding container f4af87be24685fe1e9bee397fc883ce7d13f410e8c19c6c77f167426650fb476: Status 404 returned error can't find the container with id f4af87be24685fe1e9bee397fc883ce7d13f410e8c19c6c77f167426650fb476
Sep 18 13:51:44.165255 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:44.165214    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-537abe-w7849" event={"ID":"74c81840-eb6e-4f44-8b0c-27cbcacdd8d3","Type":"ContainerStarted","Data":"fa0c5e981a968137dc5e3630b28e28b22781aab3fb47969188f07a46daa56424"}
Sep 18 13:51:44.165667 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:44.165281    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-537abe-w7849" event={"ID":"74c81840-eb6e-4f44-8b0c-27cbcacdd8d3","Type":"ContainerStarted","Data":"f4af87be24685fe1e9bee397fc883ce7d13f410e8c19c6c77f167426650fb476"}
Sep 18 13:51:44.186211 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:44.186158    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-537abe-w7849" podStartSLOduration=1.186143862 podStartE2EDuration="1.186143862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:51:44.185676134 +0000 UTC m=+2212.620552540" watchObservedRunningTime="2026-09-18 13:51:44.186143862 +0000 UTC m=+2212.621020265"
Sep 18 13:51:45.172068 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:45.172026    2566 generic.go:356] "Generic (PLEG): container finished" podID="74c81840-eb6e-4f44-8b0c-27cbcacdd8d3" containerID="fa0c5e981a968137dc5e3630b28e28b22781aab3fb47969188f07a46daa56424" exitCode=0
Sep 18 13:51:45.172458 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:45.172114    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-537abe-w7849" event={"ID":"74c81840-eb6e-4f44-8b0c-27cbcacdd8d3","Type":"ContainerDied","Data":"fa0c5e981a968137dc5e3630b28e28b22781aab3fb47969188f07a46daa56424"}
Sep 18 13:51:46.309831 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:46.309805    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-537abe-w7849"
Sep 18 13:51:46.388197 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:46.388165    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c81840-eb6e-4f44-8b0c-27cbcacdd8d3-maas-api-service-ca\") pod \"74c81840-eb6e-4f44-8b0c-27cbcacdd8d3\" (UID: \"74c81840-eb6e-4f44-8b0c-27cbcacdd8d3\") "
Sep 18 13:51:46.388551 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:46.388523    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c81840-eb6e-4f44-8b0c-27cbcacdd8d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "74c81840-eb6e-4f44-8b0c-27cbcacdd8d3" (UID: "74c81840-eb6e-4f44-8b0c-27cbcacdd8d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:51:46.489667 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:46.489547    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c81840-eb6e-4f44-8b0c-27cbcacdd8d3-maas-api-service-ca\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:51:47.185365 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:47.185333    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-537abe-w7849"
Sep 18 13:51:47.185575 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:47.185370    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-537abe-w7849" event={"ID":"74c81840-eb6e-4f44-8b0c-27cbcacdd8d3","Type":"ContainerDied","Data":"f4af87be24685fe1e9bee397fc883ce7d13f410e8c19c6c77f167426650fb476"}
Sep 18 13:51:47.185575 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:47.185417    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4af87be24685fe1e9bee397fc883ce7d13f410e8c19c6c77f167426650fb476"
Sep 18 13:51:52.558785 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:52.558752    2566 scope.go:117] "RemoveContainer" containerID="3aa4489c06eef816b16c9b481e76639238388459a806188831c5e8cd3f665643"
Sep 18 13:51:53.914571 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:53.914536    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp"]
Sep 18 13:51:53.915096 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:53.914773    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp" podUID="558b86b8-b911-4ce8-adaa-4b4af2f6833e" containerName="payload-processing" containerID="cri-o://9a54553ef9cd1564ac6f148ded3e96f9a6d2b83cc56fbe9ee57d88b90d3ccde0" gracePeriod=30
Sep 18 13:51:53.937180 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:53.937153    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9"]
Sep 18 13:51:53.937374 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:53.937353    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9" podUID="c69ee1a0-f4ea-4f73-a53a-e435c99524bb" containerName="payload-pre-processing" containerID="cri-o://4dce847a6fb991d583b478316e46aa510405e6dc15f4322c859ce5433b0dd38e" gracePeriod=30
Sep 18 13:51:54.211864 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.211835    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp"
Sep 18 13:51:54.214765 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.214743    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9"
Sep 18 13:51:54.220340 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.220299    2566 generic.go:356] "Generic (PLEG): container finished" podID="558b86b8-b911-4ce8-adaa-4b4af2f6833e" containerID="9a54553ef9cd1564ac6f148ded3e96f9a6d2b83cc56fbe9ee57d88b90d3ccde0" exitCode=0
Sep 18 13:51:54.220466 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.220345    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp"
Sep 18 13:51:54.220466 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.220378    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp" event={"ID":"558b86b8-b911-4ce8-adaa-4b4af2f6833e","Type":"ContainerDied","Data":"9a54553ef9cd1564ac6f148ded3e96f9a6d2b83cc56fbe9ee57d88b90d3ccde0"}
Sep 18 13:51:54.220466 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.220436    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp" event={"ID":"558b86b8-b911-4ce8-adaa-4b4af2f6833e","Type":"ContainerDied","Data":"c25faacb5b4944d5cf395edf2c88002e4e697f3ba36a75170167f1b4d59b57cb"}
Sep 18 13:51:54.220466 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.220463    2566 scope.go:117] "RemoveContainer" containerID="9a54553ef9cd1564ac6f148ded3e96f9a6d2b83cc56fbe9ee57d88b90d3ccde0"
Sep 18 13:51:54.221948 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.221925    2566 generic.go:356] "Generic (PLEG): container finished" podID="c69ee1a0-f4ea-4f73-a53a-e435c99524bb" containerID="4dce847a6fb991d583b478316e46aa510405e6dc15f4322c859ce5433b0dd38e" exitCode=0
Sep 18 13:51:54.222059 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.221963    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9"
Sep 18 13:51:54.222059 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.221983    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9" event={"ID":"c69ee1a0-f4ea-4f73-a53a-e435c99524bb","Type":"ContainerDied","Data":"4dce847a6fb991d583b478316e46aa510405e6dc15f4322c859ce5433b0dd38e"}
Sep 18 13:51:54.222059 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.222032    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9" event={"ID":"c69ee1a0-f4ea-4f73-a53a-e435c99524bb","Type":"ContainerDied","Data":"f5ef78929b2d396f7734bbde973fcd5ff14347ad55b1a144c3b5d512f4689aa7"}
Sep 18 13:51:54.229335 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.229316    2566 scope.go:117] "RemoveContainer" containerID="9a54553ef9cd1564ac6f148ded3e96f9a6d2b83cc56fbe9ee57d88b90d3ccde0"
Sep 18 13:51:54.229652 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:51:54.229631    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a54553ef9cd1564ac6f148ded3e96f9a6d2b83cc56fbe9ee57d88b90d3ccde0\": container with ID starting with 9a54553ef9cd1564ac6f148ded3e96f9a6d2b83cc56fbe9ee57d88b90d3ccde0 not found: ID does not exist" containerID="9a54553ef9cd1564ac6f148ded3e96f9a6d2b83cc56fbe9ee57d88b90d3ccde0"
Sep 18 13:51:54.229740 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.229665    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a54553ef9cd1564ac6f148ded3e96f9a6d2b83cc56fbe9ee57d88b90d3ccde0"} err="failed to get container status \"9a54553ef9cd1564ac6f148ded3e96f9a6d2b83cc56fbe9ee57d88b90d3ccde0\": rpc error: code = NotFound desc = could not find container \"9a54553ef9cd1564ac6f148ded3e96f9a6d2b83cc56fbe9ee57d88b90d3ccde0\": container with ID starting with 9a54553ef9cd1564ac6f148ded3e96f9a6d2b83cc56fbe9ee57d88b90d3ccde0 not found: ID does not exist"
Sep 18 13:51:54.229740 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.229690    2566 scope.go:117] "RemoveContainer" containerID="4dce847a6fb991d583b478316e46aa510405e6dc15f4322c859ce5433b0dd38e"
Sep 18 13:51:54.237898 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.237877    2566 scope.go:117] "RemoveContainer" containerID="4dce847a6fb991d583b478316e46aa510405e6dc15f4322c859ce5433b0dd38e"
Sep 18 13:51:54.238203 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:51:54.238183    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dce847a6fb991d583b478316e46aa510405e6dc15f4322c859ce5433b0dd38e\": container with ID starting with 4dce847a6fb991d583b478316e46aa510405e6dc15f4322c859ce5433b0dd38e not found: ID does not exist" containerID="4dce847a6fb991d583b478316e46aa510405e6dc15f4322c859ce5433b0dd38e"
Sep 18 13:51:54.238257 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.238209    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dce847a6fb991d583b478316e46aa510405e6dc15f4322c859ce5433b0dd38e"} err="failed to get container status \"4dce847a6fb991d583b478316e46aa510405e6dc15f4322c859ce5433b0dd38e\": rpc error: code = NotFound desc = could not find container \"4dce847a6fb991d583b478316e46aa510405e6dc15f4322c859ce5433b0dd38e\": container with ID starting with 4dce847a6fb991d583b478316e46aa510405e6dc15f4322c859ce5433b0dd38e not found: ID does not exist"
Sep 18 13:51:54.260612 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.260556    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558b86b8-b911-4ce8-adaa-4b4af2f6833e-plugins-config-volume\") pod \"558b86b8-b911-4ce8-adaa-4b4af2f6833e\" (UID: \"558b86b8-b911-4ce8-adaa-4b4af2f6833e\") "
Sep 18 13:51:54.260782 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.260706    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69ee1a0-f4ea-4f73-a53a-e435c99524bb-plugins-config-volume\") pod \"c69ee1a0-f4ea-4f73-a53a-e435c99524bb\" (UID: \"c69ee1a0-f4ea-4f73-a53a-e435c99524bb\") "
Sep 18 13:51:54.260782 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.260756    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fv75\" (UniqueName: \"kubernetes.io/projected/c69ee1a0-f4ea-4f73-a53a-e435c99524bb-kube-api-access-9fv75\") pod \"c69ee1a0-f4ea-4f73-a53a-e435c99524bb\" (UID: \"c69ee1a0-f4ea-4f73-a53a-e435c99524bb\") "
Sep 18 13:51:54.260912 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.260796    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-777p9\" (UniqueName: \"kubernetes.io/projected/558b86b8-b911-4ce8-adaa-4b4af2f6833e-kube-api-access-777p9\") pod \"558b86b8-b911-4ce8-adaa-4b4af2f6833e\" (UID: \"558b86b8-b911-4ce8-adaa-4b4af2f6833e\") "
Sep 18 13:51:54.261002 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.260972    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558b86b8-b911-4ce8-adaa-4b4af2f6833e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "558b86b8-b911-4ce8-adaa-4b4af2f6833e" (UID: "558b86b8-b911-4ce8-adaa-4b4af2f6833e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:51:54.261360 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.261317    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c69ee1a0-f4ea-4f73-a53a-e435c99524bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c69ee1a0-f4ea-4f73-a53a-e435c99524bb" (UID: "c69ee1a0-f4ea-4f73-a53a-e435c99524bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:51:54.263371 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.263346    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c69ee1a0-f4ea-4f73-a53a-e435c99524bb-kube-api-access-9fv75" (OuterVolumeSpecName: "kube-api-access-9fv75") pod "c69ee1a0-f4ea-4f73-a53a-e435c99524bb" (UID: "c69ee1a0-f4ea-4f73-a53a-e435c99524bb"). InnerVolumeSpecName "kube-api-access-9fv75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:51:54.263745 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.263721    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558b86b8-b911-4ce8-adaa-4b4af2f6833e-kube-api-access-777p9" (OuterVolumeSpecName: "kube-api-access-777p9") pod "558b86b8-b911-4ce8-adaa-4b4af2f6833e" (UID: "558b86b8-b911-4ce8-adaa-4b4af2f6833e"). InnerVolumeSpecName "kube-api-access-777p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:51:54.305137 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.305093    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-537abe-w7849"]
Sep 18 13:51:54.308871 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.308840    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-537abe-w7849"]
Sep 18 13:51:54.361767 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.361741    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c69ee1a0-f4ea-4f73-a53a-e435c99524bb-plugins-config-volume\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:51:54.361767 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.361763    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fv75\" (UniqueName: \"kubernetes.io/projected/c69ee1a0-f4ea-4f73-a53a-e435c99524bb-kube-api-access-9fv75\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:51:54.361963 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.361775    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-777p9\" (UniqueName: \"kubernetes.io/projected/558b86b8-b911-4ce8-adaa-4b4af2f6833e-kube-api-access-777p9\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:51:54.361963 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.361787    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558b86b8-b911-4ce8-adaa-4b4af2f6833e-plugins-config-volume\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:51:54.547654 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.547618    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9"]
Sep 18 13:51:54.552884 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.552859    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-537abe-7d9cdfc798-vhdq9"]
Sep 18 13:51:54.573955 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.573923    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp"]
Sep 18 13:51:54.578619 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:54.578585    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-537abe-58fc8ddcd9-2glbp"]
Sep 18 13:51:56.083498 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:56.083461    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="558b86b8-b911-4ce8-adaa-4b4af2f6833e" path="/var/lib/kubelet/pods/558b86b8-b911-4ce8-adaa-4b4af2f6833e/volumes"
Sep 18 13:51:56.083891 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:56.083825    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74c81840-eb6e-4f44-8b0c-27cbcacdd8d3" path="/var/lib/kubelet/pods/74c81840-eb6e-4f44-8b0c-27cbcacdd8d3/volumes"
Sep 18 13:51:56.084173 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:51:56.084158    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c69ee1a0-f4ea-4f73-a53a-e435c99524bb" path="/var/lib/kubelet/pods/c69ee1a0-f4ea-4f73-a53a-e435c99524bb/volumes"
Sep 18 13:52:01.869346 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:01.869307    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c50e10-rkqbv"]
Sep 18 13:52:01.869781 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:01.869770    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74c81840-eb6e-4f44-8b0c-27cbcacdd8d3" containerName="revoke-keys"
Sep 18 13:52:01.869825 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:01.869783    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="74c81840-eb6e-4f44-8b0c-27cbcacdd8d3" containerName="revoke-keys"
Sep 18 13:52:01.869825 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:01.869800    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a231903-13bc-48f0-ad95-a5ac0da21b95" containerName="istio-proxy"
Sep 18 13:52:01.869825 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:01.869805    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a231903-13bc-48f0-ad95-a5ac0da21b95" containerName="istio-proxy"
Sep 18 13:52:01.869825 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:01.869811    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="558b86b8-b911-4ce8-adaa-4b4af2f6833e" containerName="payload-processing"
Sep 18 13:52:01.869825 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:01.869817    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="558b86b8-b911-4ce8-adaa-4b4af2f6833e" containerName="payload-processing"
Sep 18 13:52:01.869991 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:01.869828    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c69ee1a0-f4ea-4f73-a53a-e435c99524bb" containerName="payload-pre-processing"
Sep 18 13:52:01.869991 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:01.869833    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c69ee1a0-f4ea-4f73-a53a-e435c99524bb" containerName="payload-pre-processing"
Sep 18 13:52:01.869991 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:01.869887    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="74c81840-eb6e-4f44-8b0c-27cbcacdd8d3" containerName="revoke-keys"
Sep 18 13:52:01.869991 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:01.869894    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c69ee1a0-f4ea-4f73-a53a-e435c99524bb" containerName="payload-pre-processing"
Sep 18 13:52:01.869991 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:01.869904    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="558b86b8-b911-4ce8-adaa-4b4af2f6833e" containerName="payload-processing"
Sep 18 13:52:01.869991 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:01.869910    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a231903-13bc-48f0-ad95-a5ac0da21b95" containerName="istio-proxy"
Sep 18 13:52:01.872940 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:01.872924    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c50e10-rkqbv"
Sep 18 13:52:01.875587 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:01.875563    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nh2cd\""
Sep 18 13:52:01.882484 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:01.876556    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:52:01.934041 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:01.933997    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4848ba34-afcd-41c2-8f96-f3c50d8a4683-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c50e10-rkqbv\" (UID: \"4848ba34-afcd-41c2-8f96-f3c50d8a4683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c50e10-rkqbv"
Sep 18 13:52:01.938647 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:01.938618    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c50e10-rkqbv"]
Sep 18 13:52:02.035502 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:02.035454    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4848ba34-afcd-41c2-8f96-f3c50d8a4683-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c50e10-rkqbv\" (UID: \"4848ba34-afcd-41c2-8f96-f3c50d8a4683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c50e10-rkqbv"
Sep 18 13:52:02.036143 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:02.036121    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4848ba34-afcd-41c2-8f96-f3c50d8a4683-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c50e10-rkqbv\" (UID: \"4848ba34-afcd-41c2-8f96-f3c50d8a4683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c50e10-rkqbv"
Sep 18 13:52:02.190105 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:02.190017    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c50e10-rkqbv"
Sep 18 13:52:02.464304 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:02.464277    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c50e10-rkqbv"]
Sep 18 13:52:02.466508 ip-10-0-129-234 kubenswrapper[2566]: W0918 13:52:02.466477    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4848ba34_afcd_41c2_8f96_f3c50d8a4683.slice/crio-1f7d35583d918ebef9c515f85b8d5fb717378cc66eed11fcc2bac2ba1a4854dc WatchSource:0}: Error finding container 1f7d35583d918ebef9c515f85b8d5fb717378cc66eed11fcc2bac2ba1a4854dc: Status 404 returned error can't find the container with id 1f7d35583d918ebef9c515f85b8d5fb717378cc66eed11fcc2bac2ba1a4854dc
Sep 18 13:52:03.275001 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:03.274965    2566 generic.go:356] "Generic (PLEG): container finished" podID="4848ba34-afcd-41c2-8f96-f3c50d8a4683" containerID="257fb08d6df117b4a2a393baebbf6a2a71f747b7dcaa1a654dde08ad730c54ef" exitCode=0
Sep 18 13:52:03.275433 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:03.275051    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c50e10-rkqbv" event={"ID":"4848ba34-afcd-41c2-8f96-f3c50d8a4683","Type":"ContainerDied","Data":"257fb08d6df117b4a2a393baebbf6a2a71f747b7dcaa1a654dde08ad730c54ef"}
Sep 18 13:52:03.275433 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:03.275105    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c50e10-rkqbv" event={"ID":"4848ba34-afcd-41c2-8f96-f3c50d8a4683","Type":"ContainerStarted","Data":"1f7d35583d918ebef9c515f85b8d5fb717378cc66eed11fcc2bac2ba1a4854dc"}
Sep 18 13:52:04.405956 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:04.405927    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c50e10-rkqbv"
Sep 18 13:52:04.556997 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:04.556906    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4848ba34-afcd-41c2-8f96-f3c50d8a4683-maas-api-service-ca\") pod \"4848ba34-afcd-41c2-8f96-f3c50d8a4683\" (UID: \"4848ba34-afcd-41c2-8f96-f3c50d8a4683\") "
Sep 18 13:52:04.557259 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:04.557236    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4848ba34-afcd-41c2-8f96-f3c50d8a4683-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4848ba34-afcd-41c2-8f96-f3c50d8a4683" (UID: "4848ba34-afcd-41c2-8f96-f3c50d8a4683"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:52:04.658473 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:04.658439    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4848ba34-afcd-41c2-8f96-f3c50d8a4683-maas-api-service-ca\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:52:05.287028 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:05.286987    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c50e10-rkqbv" event={"ID":"4848ba34-afcd-41c2-8f96-f3c50d8a4683","Type":"ContainerDied","Data":"1f7d35583d918ebef9c515f85b8d5fb717378cc66eed11fcc2bac2ba1a4854dc"}
Sep 18 13:52:05.287200 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:05.287040    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f7d35583d918ebef9c515f85b8d5fb717378cc66eed11fcc2bac2ba1a4854dc"
Sep 18 13:52:05.287200 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:05.287000    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c50e10-rkqbv"
Sep 18 13:52:12.376478 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:12.376432    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c50e10-29828985-6snkz"]
Sep 18 13:52:12.382067 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:12.382036    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c50e10-29828985-6snkz"]
Sep 18 13:52:12.419197 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:12.419161    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb"]
Sep 18 13:52:12.419391 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:12.419365    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb" podUID="d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b" containerName="payload-processing" containerID="cri-o://aa778e157082ef1a68e4294f41ba78865eadeb40440794217e276fec59af76da" gracePeriod=30
Sep 18 13:52:12.669693 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:12.669651    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb"
Sep 18 13:52:12.759906 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:12.759869    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c50e10-rkqbv"]
Sep 18 13:52:12.762967 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:12.762941    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c50e10-rkqbv"]
Sep 18 13:52:12.829685 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:12.829656    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b-plugins-config-volume\") pod \"d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b\" (UID: \"d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b\") "
Sep 18 13:52:12.829847 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:12.829701    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5kpj\" (UniqueName: \"kubernetes.io/projected/d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b-kube-api-access-g5kpj\") pod \"d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b\" (UID: \"d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b\") "
Sep 18 13:52:12.830120 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:12.830100    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b" (UID: "d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:52:12.832009 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:12.831980    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b-kube-api-access-g5kpj" (OuterVolumeSpecName: "kube-api-access-g5kpj") pod "d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b" (UID: "d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b"). InnerVolumeSpecName "kube-api-access-g5kpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:52:12.930991 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:12.930894    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b-plugins-config-volume\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:52:12.930991 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:12.930932    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5kpj\" (UniqueName: \"kubernetes.io/projected/d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b-kube-api-access-g5kpj\") on node \"ip-10-0-129-234.ec2.internal\" DevicePath \"\""
Sep 18 13:52:13.330466 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:13.330374    2566 generic.go:356] "Generic (PLEG): container finished" podID="d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b" containerID="aa778e157082ef1a68e4294f41ba78865eadeb40440794217e276fec59af76da" exitCode=0
Sep 18 13:52:13.330466 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:13.330436    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb"
Sep 18 13:52:13.330701 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:13.330461    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb" event={"ID":"d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b","Type":"ContainerDied","Data":"aa778e157082ef1a68e4294f41ba78865eadeb40440794217e276fec59af76da"}
Sep 18 13:52:13.330701 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:13.330510    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb" event={"ID":"d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b","Type":"ContainerDied","Data":"bdba9f2fb8a93afb8bd9d1fbf7a5f42ed307a3000cbdb87544138018d6fbb3f8"}
Sep 18 13:52:13.330701 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:13.330531    2566 scope.go:117] "RemoveContainer" containerID="aa778e157082ef1a68e4294f41ba78865eadeb40440794217e276fec59af76da"
Sep 18 13:52:13.339978 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:13.339957    2566 scope.go:117] "RemoveContainer" containerID="aa778e157082ef1a68e4294f41ba78865eadeb40440794217e276fec59af76da"
Sep 18 13:52:13.340233 ip-10-0-129-234 kubenswrapper[2566]: E0918 13:52:13.340213    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa778e157082ef1a68e4294f41ba78865eadeb40440794217e276fec59af76da\": container with ID starting with aa778e157082ef1a68e4294f41ba78865eadeb40440794217e276fec59af76da not found: ID does not exist" containerID="aa778e157082ef1a68e4294f41ba78865eadeb40440794217e276fec59af76da"
Sep 18 13:52:13.340281 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:13.340241    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa778e157082ef1a68e4294f41ba78865eadeb40440794217e276fec59af76da"} err="failed to get container status \"aa778e157082ef1a68e4294f41ba78865eadeb40440794217e276fec59af76da\": rpc error: code = NotFound desc = could not find container \"aa778e157082ef1a68e4294f41ba78865eadeb40440794217e276fec59af76da\": container with ID starting with aa778e157082ef1a68e4294f41ba78865eadeb40440794217e276fec59af76da not found: ID does not exist"
Sep 18 13:52:13.357250 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:13.357221    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb"]
Sep 18 13:52:13.362719 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:13.362684    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c50e10-b9d67bd9c-49rmb"]
Sep 18 13:52:14.076412 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:14.076378    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c43d049-12b7-4c6f-ba03-d7b3895a9e31" path="/var/lib/kubelet/pods/2c43d049-12b7-4c6f-ba03-d7b3895a9e31/volumes"
Sep 18 13:52:14.076798 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:14.076708    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4848ba34-afcd-41c2-8f96-f3c50d8a4683" path="/var/lib/kubelet/pods/4848ba34-afcd-41c2-8f96-f3c50d8a4683/volumes"
Sep 18 13:52:14.077027 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:14.077013    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b" path="/var/lib/kubelet/pods/d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b/volumes"
Sep 18 13:52:52.585261 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:52:52.585220    2566 scope.go:117] "RemoveContainer" containerID="8515657b47b25c1ba428a20da4e96cb27dcc5d9f3fd2c961fbbf122adb7c7c9f"
Sep 18 13:53:52.616946 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:53:52.616865    2566 scope.go:117] "RemoveContainer" containerID="b347569927b6024adeccbc57cb59ab716501c94c98855954a059d1c5cf527f1b"
Sep 18 13:54:52.203255 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:54:52.203220    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k86vn_55e33e14-3f7a-4793-a724-9fd5022740d7/console-operator/1.log"
Sep 18 13:54:52.209098 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:54:52.209068    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k86vn_55e33e14-3f7a-4793-a724-9fd5022740d7/console-operator/1.log"
Sep 18 13:56:52.653403 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:56:52.653328    2566 scope.go:117] "RemoveContainer" containerID="b75e49238152b3c9e50b59a5c6a14a7d4ae9d2fae9e916c192597b5aba660152"
Sep 18 13:57:52.672494 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:57:52.672462    2566 scope.go:117] "RemoveContainer" containerID="fa0c5e981a968137dc5e3630b28e28b22781aab3fb47969188f07a46daa56424"
Sep 18 13:58:47.782322 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:58:47.782291    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-sx4ff_9b4bc476-98f6-4663-a919-1351827790dd/server/0.log"
Sep 18 13:58:47.912923 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:58:47.912888    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-zc529_05bcebef-a0aa-46ec-9bf5-ca9c83c1770b/manager/0.log"
Sep 18 13:58:48.280050 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:58:48.280020    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-854585bc8d-zllbr_ddec8697-ca0a-4402-89cb-f0707b414d64/copy-manifests/0.log"
Sep 18 13:58:48.289582 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:58:48.289553    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-854585bc8d-zllbr_ddec8697-ca0a-4402-89cb-f0707b414d64/manager/0.log"
Sep 18 13:58:48.878169 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:58:48.878139    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29828970-nfv72_4511b800-23d4-457b-a5bc-99c3d2605431/cleanup/0.log"
Sep 18 13:58:51.028656 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:58:51.028626    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4_a6530f20-b9e0-4f5b-bae3-cc99a01845e4/istio-proxy/0.log"
Sep 18 13:58:51.156613 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:58:51.156572    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8lgh8_a8c4f346-cfd7-4e0d-b482-2f670e7817d4/discovery/0.log"
Sep 18 13:58:51.274284 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:58:51.274254    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c5fc7c97f-f7z25_7399cabd-9c15-41c4-aef1-cdc23e893e43/kube-auth-proxy/0.log"
Sep 18 13:58:52.692767 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:58:52.692726    2566 scope.go:117] "RemoveContainer" containerID="257fb08d6df117b4a2a393baebbf6a2a71f747b7dcaa1a654dde08ad730c54ef"
Sep 18 13:59:01.489191 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:01.489160    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bgp65_15a0691d-d5b4-41d9-a89c-705ee52f7b25/global-pull-secret-syncer/0.log"
Sep 18 13:59:01.705886 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:01.705854    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-b8g55_df010623-6a98-476b-9bd6-26146ad788c8/konnectivity-agent/0.log"
Sep 18 13:59:01.772415 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:01.772342    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-234.ec2.internal_d6137ead809380c4ef87cd83540755ba/haproxy/0.log"
Sep 18 13:59:08.695190 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:08.695151    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ebdcd5ef-4bda-4dd2-a279-6d96c37d011d/alertmanager/0.log"
Sep 18 13:59:08.722270 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:08.722243    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ebdcd5ef-4bda-4dd2-a279-6d96c37d011d/config-reloader/0.log"
Sep 18 13:59:08.767965 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:08.767882    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ebdcd5ef-4bda-4dd2-a279-6d96c37d011d/kube-rbac-proxy-web/0.log"
Sep 18 13:59:08.794432 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:08.794406    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ebdcd5ef-4bda-4dd2-a279-6d96c37d011d/kube-rbac-proxy/0.log"
Sep 18 13:59:08.830790 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:08.830763    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ebdcd5ef-4bda-4dd2-a279-6d96c37d011d/kube-rbac-proxy-metric/0.log"
Sep 18 13:59:08.856470 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:08.856443    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ebdcd5ef-4bda-4dd2-a279-6d96c37d011d/prom-label-proxy/0.log"
Sep 18 13:59:08.885869 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:08.885831    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ebdcd5ef-4bda-4dd2-a279-6d96c37d011d/init-config-reloader/0.log"
Sep 18 13:59:08.927231 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:08.927201    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-pg4dm_6b914660-2e06-4c5f-897f-f3ac5b12c348/cluster-monitoring-operator/0.log"
Sep 18 13:59:08.954237 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:08.954207    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-9d24f_bdc18c51-ef7d-45ee-9e5c-7805dfcf618c/kube-state-metrics/0.log"
Sep 18 13:59:08.980217 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:08.980185    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-9d24f_bdc18c51-ef7d-45ee-9e5c-7805dfcf618c/kube-rbac-proxy-main/0.log"
Sep 18 13:59:09.005643 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:09.005587    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-9d24f_bdc18c51-ef7d-45ee-9e5c-7805dfcf618c/kube-rbac-proxy-self/0.log"
Sep 18 13:59:09.201194 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:09.201164    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tf756_6c168813-ac20-48b4-b2a3-48664b28466a/node-exporter/0.log"
Sep 18 13:59:09.226520 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:09.226492    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tf756_6c168813-ac20-48b4-b2a3-48664b28466a/kube-rbac-proxy/0.log"
Sep 18 13:59:09.250956 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:09.250932    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tf756_6c168813-ac20-48b4-b2a3-48664b28466a/init-textfile/0.log"
Sep 18 13:59:09.727503 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:09.727472    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bc57df576-ffwq4_147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0/telemeter-client/0.log"
Sep 18 13:59:09.763742 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:09.763712    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bc57df576-ffwq4_147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0/reload/0.log"
Sep 18 13:59:09.788747 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:09.788718    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bc57df576-ffwq4_147f2b4c-f4a2-4be5-a57b-6b098a8c8dd0/kube-rbac-proxy/0.log"
Sep 18 13:59:11.297643 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.297588    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq"]
Sep 18 13:59:11.298163 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.298138    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b" containerName="payload-processing"
Sep 18 13:59:11.298273 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.298164    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b" containerName="payload-processing"
Sep 18 13:59:11.298273 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.298180    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4848ba34-afcd-41c2-8f96-f3c50d8a4683" containerName="revoke-keys"
Sep 18 13:59:11.298273 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.298190    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4848ba34-afcd-41c2-8f96-f3c50d8a4683" containerName="revoke-keys"
Sep 18 13:59:11.298405 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.298277    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4848ba34-afcd-41c2-8f96-f3c50d8a4683" containerName="revoke-keys"
Sep 18 13:59:11.298405 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.298287    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5f7ad61-fa71-4bbe-93c0-f10f7d5a871b" containerName="payload-processing"
Sep 18 13:59:11.301272 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.301255    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq"
Sep 18 13:59:11.303674 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.303642    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gnq97\"/\"openshift-service-ca.crt\""
Sep 18 13:59:11.303826 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.303673    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gnq97\"/\"default-dockercfg-jvgfk\""
Sep 18 13:59:11.304369 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.304347    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gnq97\"/\"kube-root-ca.crt\""
Sep 18 13:59:11.312191 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.312167    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq"]
Sep 18 13:59:11.362242 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.362209    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2c2da9f6-fdcc-4409-a79c-1c3e2417df63-proc\") pod \"perf-node-gather-daemonset-bmkxq\" (UID: \"2c2da9f6-fdcc-4409-a79c-1c3e2417df63\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq"
Sep 18 13:59:11.362426 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.362256    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gz2t\" (UniqueName: \"kubernetes.io/projected/2c2da9f6-fdcc-4409-a79c-1c3e2417df63-kube-api-access-7gz2t\") pod \"perf-node-gather-daemonset-bmkxq\" (UID: \"2c2da9f6-fdcc-4409-a79c-1c3e2417df63\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq"
Sep 18 13:59:11.362426 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.362285    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2c2da9f6-fdcc-4409-a79c-1c3e2417df63-podres\") pod \"perf-node-gather-daemonset-bmkxq\" (UID: \"2c2da9f6-fdcc-4409-a79c-1c3e2417df63\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq"
Sep 18 13:59:11.362426 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.362343    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c2da9f6-fdcc-4409-a79c-1c3e2417df63-sys\") pod \"perf-node-gather-daemonset-bmkxq\" (UID: \"2c2da9f6-fdcc-4409-a79c-1c3e2417df63\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq"
Sep 18 13:59:11.362426 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.362363    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c2da9f6-fdcc-4409-a79c-1c3e2417df63-lib-modules\") pod \"perf-node-gather-daemonset-bmkxq\" (UID: \"2c2da9f6-fdcc-4409-a79c-1c3e2417df63\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq"
Sep 18 13:59:11.463715 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.463669    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2c2da9f6-fdcc-4409-a79c-1c3e2417df63-podres\") pod \"perf-node-gather-daemonset-bmkxq\" (UID: \"2c2da9f6-fdcc-4409-a79c-1c3e2417df63\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq"
Sep 18 13:59:11.463898 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.463743    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c2da9f6-fdcc-4409-a79c-1c3e2417df63-sys\") pod \"perf-node-gather-daemonset-bmkxq\" (UID: \"2c2da9f6-fdcc-4409-a79c-1c3e2417df63\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq"
Sep 18 13:59:11.463898 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.463765    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c2da9f6-fdcc-4409-a79c-1c3e2417df63-lib-modules\") pod \"perf-node-gather-daemonset-bmkxq\" (UID: \"2c2da9f6-fdcc-4409-a79c-1c3e2417df63\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq"
Sep 18 13:59:11.463898 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.463813    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2c2da9f6-fdcc-4409-a79c-1c3e2417df63-proc\") pod \"perf-node-gather-daemonset-bmkxq\" (UID: \"2c2da9f6-fdcc-4409-a79c-1c3e2417df63\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq"
Sep 18 13:59:11.463898 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.463834    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c2da9f6-fdcc-4409-a79c-1c3e2417df63-sys\") pod \"perf-node-gather-daemonset-bmkxq\" (UID: \"2c2da9f6-fdcc-4409-a79c-1c3e2417df63\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq"
Sep 18 13:59:11.463898 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.463852    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2c2da9f6-fdcc-4409-a79c-1c3e2417df63-podres\") pod \"perf-node-gather-daemonset-bmkxq\" (UID: \"2c2da9f6-fdcc-4409-a79c-1c3e2417df63\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq"
Sep 18 13:59:11.463898 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.463848    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gz2t\" (UniqueName: \"kubernetes.io/projected/2c2da9f6-fdcc-4409-a79c-1c3e2417df63-kube-api-access-7gz2t\") pod \"perf-node-gather-daemonset-bmkxq\" (UID: \"2c2da9f6-fdcc-4409-a79c-1c3e2417df63\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq"
Sep 18 13:59:11.463898 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.463877    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c2da9f6-fdcc-4409-a79c-1c3e2417df63-lib-modules\") pod \"perf-node-gather-daemonset-bmkxq\" (UID: \"2c2da9f6-fdcc-4409-a79c-1c3e2417df63\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq"
Sep 18 13:59:11.464220 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.463931    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2c2da9f6-fdcc-4409-a79c-1c3e2417df63-proc\") pod \"perf-node-gather-daemonset-bmkxq\" (UID: \"2c2da9f6-fdcc-4409-a79c-1c3e2417df63\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq"
Sep 18 13:59:11.472391 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.472363    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gz2t\" (UniqueName: \"kubernetes.io/projected/2c2da9f6-fdcc-4409-a79c-1c3e2417df63-kube-api-access-7gz2t\") pod \"perf-node-gather-daemonset-bmkxq\" (UID: \"2c2da9f6-fdcc-4409-a79c-1c3e2417df63\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq"
Sep 18 13:59:11.612392 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.612357    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq"
Sep 18 13:59:11.845649 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.845614    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k86vn_55e33e14-3f7a-4793-a724-9fd5022740d7/console-operator/1.log"
Sep 18 13:59:11.850660 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.850638    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-k86vn_55e33e14-3f7a-4793-a724-9fd5022740d7/console-operator/2.log"
Sep 18 13:59:11.887741 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:11.887717    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq"]
Sep 18 13:59:12.353136 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:12.353102    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq" event={"ID":"2c2da9f6-fdcc-4409-a79c-1c3e2417df63","Type":"ContainerStarted","Data":"a91cf57155115b6f584288956cc89a8c78ac6e27258b00130aef33e920cde1cf"}
Sep 18 13:59:12.353543 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:12.353157    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq" event={"ID":"2c2da9f6-fdcc-4409-a79c-1c3e2417df63","Type":"ContainerStarted","Data":"a06e1b41e835c44bdedafd87a2aacde2f970aa4b89746d68fc46641dfc204710"}
Sep 18 13:59:12.353543 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:12.353251    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq"
Sep 18 13:59:12.370140 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:12.370086    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq" podStartSLOduration=1.370071259 podStartE2EDuration="1.370071259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:12.369167621 +0000 UTC m=+2660.804044045" watchObservedRunningTime="2026-09-18 13:59:12.370071259 +0000 UTC m=+2660.804947663"
Sep 18 13:59:12.420686 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:12.420655    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-vtcgz_e167d2a4-7a91-46b1-a5e3-da0a27ede351/download-server/0.log"
Sep 18 13:59:13.100982 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:13.100955    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-pnzr7_cfad07b4-8ee8-4ce6-81bf-5bdc3f601aa0/volume-data-source-validator/0.log"
Sep 18 13:59:14.532146 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:14.532113    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r94bv_96cd73fd-fcb1-44ea-b184-63a97a0277f4/dns/0.log"
Sep 18 13:59:14.564059 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:14.564031    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r94bv_96cd73fd-fcb1-44ea-b184-63a97a0277f4/kube-rbac-proxy/0.log"
Sep 18 13:59:14.689198 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:14.689162    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rd9fl_c6e79ce7-2e57-4103-871f-5b11f0a468d5/dns-node-resolver/0.log"
Sep 18 13:59:15.288740 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:15.288709    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lglgb_df2366e7-fd3e-49d3-94af-686a9a9f262f/node-ca/0.log"
Sep 18 13:59:16.409563 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:16.409532    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4_a6530f20-b9e0-4f5b-bae3-cc99a01845e4/istio-proxy/0.log"
Sep 18 13:59:16.843434 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:16.843400    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8lgh8_a8c4f346-cfd7-4e0d-b482-2f670e7817d4/discovery/0.log"
Sep 18 13:59:16.865288 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:16.865263    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c5fc7c97f-f7z25_7399cabd-9c15-41c4-aef1-cdc23e893e43/kube-auth-proxy/0.log"
Sep 18 13:59:17.987252 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:17.987226    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cn6sm_2452828a-bce2-4da3-a477-88d9d048a4b3/serve-healthcheck-canary/0.log"
Sep 18 13:59:18.368798 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:18.368768    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-bmkxq"
Sep 18 13:59:18.919155 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:18.919126    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xjrc4_3d630003-2bfd-4cb9-9b8a-745c0ccca548/kube-rbac-proxy/0.log"
Sep 18 13:59:18.946126 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:18.946100    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xjrc4_3d630003-2bfd-4cb9-9b8a-745c0ccca548/exporter/0.log"
Sep 18 13:59:18.973983 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:18.973958    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xjrc4_3d630003-2bfd-4cb9-9b8a-745c0ccca548/extractor/0.log"
Sep 18 13:59:21.861335 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:21.861303    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-sx4ff_9b4bc476-98f6-4663-a919-1351827790dd/server/0.log"
Sep 18 13:59:21.955604 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:21.955555    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-zc529_05bcebef-a0aa-46ec-9bf5-ca9c83c1770b/manager/0.log"
Sep 18 13:59:22.142125 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:22.142096    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4_a6530f20-b9e0-4f5b-bae3-cc99a01845e4/istio-proxy/0.log"
Sep 18 13:59:22.152721 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:22.152699    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-854585bc8d-zllbr_ddec8697-ca0a-4402-89cb-f0707b414d64/manager/0.log"
Sep 18 13:59:22.180939 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:22.180911    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-854585bc8d-zllbr_ddec8697-ca0a-4402-89cb-f0707b414d64/copy-manifests/0.log"
Sep 18 13:59:22.584352 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:22.584301    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8lgh8_a8c4f346-cfd7-4e0d-b482-2f670e7817d4/discovery/0.log"
Sep 18 13:59:22.595425 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:22.595399    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c5fc7c97f-f7z25_7399cabd-9c15-41c4-aef1-cdc23e893e43/kube-auth-proxy/0.log"
Sep 18 13:59:23.533284 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:23.533253    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-85985676b5-4gsh6_7ed304aa-a520-4060-ab6a-f706761b322c/manager/0.log"
Sep 18 13:59:24.287640 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:24.287585    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-xk9c4_a6530f20-b9e0-4f5b-bae3-cc99a01845e4/istio-proxy/0.log"
Sep 18 13:59:24.712699 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:24.712669    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-8lgh8_a8c4f346-cfd7-4e0d-b482-2f670e7817d4/discovery/0.log"
Sep 18 13:59:24.757291 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:24.757256    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c5fc7c97f-f7z25_7399cabd-9c15-41c4-aef1-cdc23e893e43/kube-auth-proxy/0.log"
Sep 18 13:59:29.262746 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:29.262710    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-c9tj5_1555f97d-9829-4708-b2c7-b2afb963ad05/kube-storage-version-migrator-operator/1.log"
Sep 18 13:59:29.263620 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:29.263589    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-c9tj5_1555f97d-9829-4708-b2c7-b2afb963ad05/kube-storage-version-migrator-operator/0.log"
Sep 18 13:59:30.409563 ip-10-0-129-234 kubenswrapper[2566]: I0918 13:59:30.409533    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4pn75_06c94dfa-d0af-4756-9cb6-65795af43490/kube-multus/0.log"